DOCUMENT RESUME 



ED 235 314 



CE 037 052 



AUTHOR 
TITLE 

INSTITUTION 
SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Dodson, Richard; Kogan, Deborah, Ed. 

Statistical Analysis in Evaluation Research: Tools 

for Investigating Problems in VR Performance. 

Berkeley Planning Associates, Calif. 

Rehabilitation Services Administration (ED) , 

Washington , DC. 

Apr 83 

300-81-0435; HEW-105-79-4000 

98p.; This report is a revised version of a chapter 
included in an original draft of a report entitled 
"Analytic Paradigm for the Revised Vocational 
Rehabilitation Program Standards" (July 1981). 
Reports - Research/Technical (143) 

MF01/PCC4 Plus Postage. 

Adult Vocational Education; Performance; *Program 
Evaluation; *Standards; State Agencies; *Statistical 
Analysis; *Vocational Rehabilitation 



ABSTRACT 

This report reviews the ways in which statistica3. 
analysis can be used as a tool by vocational rehabilitation program 
managers to investigate the causes of problematic performance and 
generate strategies for corrective action. Two types of, data 
collection are noted: operational studies and statistical data 
studies. Descriptions follow of two levels of analysis: client level 
(micro level) and site or agency level (macro level). Two functions 
fulfilled by client-level analyses are discussed: (1) checking the 
validity, reliability, and accuracy of the information used in 
standards and informational data elements and (2) following up the 
program managers 1 investigations and the results of the agency-level 
analysis to explore the detailed interaction between client 
characteristics, service interventions, and observed client outcomes. 
Each description of an approach is followed by a case example. Two 
questions for evaluative research on the standards that are addressed 
by agency-level analyses are then considered: (1) how is attainment 
on the various standards data elements related across the state 
vocational rehabilitation agencies and (2) what explains the 
differences among state vocational rehabilitation agencies in 
attainment on specific standards data elements? A case example 
follows each description of an approach. (YLB) 
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ABSTRACT 



STATISTICAL ANALYSIS IN EVALUATION RESEARCH: TOOLS FOR INVESTIGATING 
PROBLEMS IN VR PERFORMANCE , by Richard Dodson, Ph.D., edited by Deborah 
Kogan, April 1983, 92 pp. 

This report reviews the ways in which statistical analysis can be 
utilized as a tool by VR program managers to investigate the causes of 
problematic performance and generate strategies for corrective action. 
Two levels of analysis client-level ("micro"-level) analysis and site- 
or agency-level ( n macro"-level) analysis — are described; several case 
examples of each type of evaluation research are provided. 
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I. INTRODUCTION 

THE PROGRAM STANDARDS EVALUATION SYSTEM 

The 1973 Renabilitation Act contained, among its many other provisions., 
a requirement that evaluation standards be devised and implemented to meas- 
ure the performance of the VR program in achieving its mandate. Between 
1978 and 1982, Berkeley Planning Associates (BPA) , under contract to the 
Rehabilitation Services Administration (RSA) , jdeveloped an integrated stan- 
dards evaluation system. The proposed Program Standards Evaluation System 
evaluates the federal-state VR programs. The proposed program evaluation 
standards include eight performance standards and associated, data elements 
and five procedural standards and associated data elements. The performance 
standards pertain to service outcomes (productivity, effectiveness, impact) 
while the procedural standards pertain to service method and process (e.g., 
case handling, data quality). 

The program standards system is a system for evaluating and managing 
parts of the VR system. The standards are intended to support state agency 
decision-making and point to specific actions for program improvement or 
change. 

The purposes of the program standards system are, simply: 

e to make available information on the achievement of state 
VR agencies with respect to VR goals as measured by the 
standards data elements; and, more importantly, 

o to guide the behavior of state VR agencies toward greater 
achievement on those standards 1 data elements; as well as 

• to identify possible problems and corrective actions, when- 
ever state VR agencies are unable to reach their achievement 
objectives. 

What is new about the revised standards system is that it is oriented 
to guiding the behavior of the state VR agencies in new directions, not 
just reporting on past behavior. The revised standards system is pros- 
pective, not retrospective, oriented to suggesting directions for future 
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behavior and not just to reporting on past behavior. By setting objectives 
for each state VR agency to achieve on each of the standards data elements, 
the VR system can be guided in the directions that RSA and the states want 
to go. The. overall direction of the VR program thus can be changed, as 
can the achievement of particular state VR agencies. 

The paradigm of the system is concerned with flagging problematic 
attainment, investigating possible solutions, and identifying corrective 
actions. 

What is also very unique about the revised standards system is that 
it does not stop when a state VR agency fails to meet its objective on a 
particular standard data element. Instead, in the revised standards system, 
the decision support system identifies possible problems and corrective 
actions. This system is designed to enable program managers to quickly 
identify whether possible problems can be identified or whether further 
evaluation is required. 

INVESTIGATING PROBLEMATIC ATTAINMENT 

According to the analytic paradigm for the program standards system, 
a state's performance on the data elements should be compared to the per- 
formance levels set for that period. Some agencies will not have met some 
of their objectives set for level of attainment on the standards. The system 
does not stop with this comparison or grading, but instead moves to investi- 
gate the problematic attainment and to identify corrective actions as part 
of the decision support system . The purpose of the decision support system 
is to close the gap between reporting on the standards and actions based 
on the standards. The system should: 

• provide an ability to pinpoint causes for problems in attainment; 

• identify strategies leading to enhanced attainment; and 

s identify appropriate policy recommendations and program actions 
that can be taken by state agencies, RSA, or Congress, based on 
the analysis and aimed at improvement in agency attainment . 

In order to identify the causes of problematic attainment and develop 
strategies for corrective action, program managers must be provided with 
information that is: 
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• relevant to the issues (ie., problems) under consideration; 

• quickly and easily interpretable; 

• timely; and 

o suggestive either of an immediatf policy response to the problem, 
or of further investigation needed before an appropriate response 
can be formulated. 

Sometimes this information can be developed directly by program staff 
using data from the standards reporting system, plus agency knowledge of 
program operations. At other times, the information needed to identify the 
causes of problems and ways to correct these problems must be developed 
through a more extensive process of evaluation research , carried out by 
evaluation staff or by outside consultants, using agency data and other 
data bases or on-site investigations and case reviews, as needed. 

This report is intended to provide state agencies and others with a 
better understanding of how statistical analysis^ techniques can be applied 
in the process of problem identification and corrective action. Data sources 
for the statistical analysis in such evaluation research will be many and 
varied. First are those sources resulting from routine reporting within the 
program. These include: 

• the R-300 (or other client record); 

• case reviews; 

• closure and follow-up surveys; 

• the agency-level standard statistical reporting forms; 

• caseload statistics; 

© summaries of agency organization types, resources, internal 
procedures, and service provision patterns; and 

• the MIS and FMIS. 

In addition to routine program reporting, other important data sources 
include information from other federal agencies and departments, and special 
studies conducted by RSA or by contract research. For example, the Depart- 
ments of Labor and Commerce may provide useful information on national 
economic trends and labor market conditions. Special studies may be 
conducted for several reasons: to generate new knowledge on variables 
hypothesized to impact on program success; to further study or validate 
the relationship between program success and independent factors that 
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already have been observed to impact on program success; or to update 
and/or provide information needed to test a "full" causal model of program 
success (examples here include the needs for data on client motivation 
and "counselor effort" by client). 

For the most part, agency-level evaluation research will rely on 
already-published data, usually based on the full population of VR clients. 
Data sources here include the program data book and information on client 
characteristics and caseload statistics. Aggregate data on performance 
will be merged with information from other program reports (e.g., on costs, 
numbers served, services offered, and case reviews) and from data received 
from non-RSA sources (e.g., on economic and labor market trends). This 
pool of aggregated data will provide the bulk of information used in the 
agency-level evaluation research. 

Evaluation research may require some analysis that calls for client . 
data that are not regularly collected. If this is the case, such additional 
data collection should be done on a smaller sample of clients. Part of the 
data collection for the standards data elements is already based on samples 
of clients, e.g., the closure survey, the case review. New data collection 
to support a client-level analysis would be on a one-time-only, as-needed 
basis and not part of the data collection system. 

Evaluation research is intended to supplement the reports of state VR 
agency attainment on the standards. After such reports have been submitted, 
they will undergo review by program evaluators and managers who will make 
preliminary determinations of the necessity for evaluation research. 

TYPES AND LEVELS OF ANALYSIS FOR EVALUATION RESEARCH 

The most important question for the evaluation research component 
concerns the types and levels of analysis for the research. First, two 
types of date collection and of analysis operational studies and statis- 
tical data studies -- are noted. Then, "micro" and "macro" levels of statis- 
tical data analysis are described. 
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Types of Data Collection and of Analysis: Operational Studies versus 
Statistical Data Studies 

So far, ail of the data collection that has been described has been 
of a "quantitative 11 nature, which is usually associated with evaluation 
research. However, there is a whole otJier style of data collection that 
is useful in evaluation research that is more qualitiative . No attempt 
will be made here to specify the myriad types of qualitative data collection 
— there is a vast literature on the subject (see e.g., Cook and Reichardt, 
Bogdan and Taylor, or Douglas). The techniques of participant observa- 
tion, of unstructured interviewing, or of investigative social research 
could all be used to identify possible problems and corrective actions. 
In the VR field, there is already a structure for such data collection 
around the PARs or SMARs. The point is that in the face of increasing 
data processing capabilities, the very important role of qualitative infor- 
mation must not be lost. 

These operational studies can be used to generate hypotheses to be 
tested via statistical data analyses, or can be used to understand the 
findings from the statistical data analyses. Some of the qualitative 
information collected in the operational studies could be turned into 
quantitative data, usually at the ordinal or nominal level of measurement. 

Levels of Statistical Data Analysis 

One of the first choices that must be made in deciding on the kinds 
of statistical data analysis to do is the choice of a level of analysis. 
Once the major choice between a "micro" and "macro" level of analysis is 
made, the analyst chooses the unit of analysis. Below, the difference 
between these two types of analysis is described. Then, the relative 
advantages and disadvantages of each level of analysis are presented. 

,f Micro" and "Macro" 

The difference between a micro (client-level) and a macro (service 
system level) of analysis is primarily a function of tne perspective of 
the analyst. In economics, microanalysis focuses on the individual con- 
sumer when applied to an understanding of consumption. Alternatively, a 
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micro level of analysis for understanding production focuses on the firm, 
a much larger unit of analysis. While one analyst's micro level of analysis 
may thus be another analyst's macro level of analysis, a crude but simple 
across-the-board distinction between the two levels is helpful . Simply, a 
micro level of analysis is at the level of the smallest possible unit of 
observation and analysis, while a macro level of analysis is aggregated 
or combined to form some larger unit of analysis, subsuming many of the 
micro units of observation. 

For social service delivery systems, at the micro level of analysis 
the unit of observation and analysis is usually the individual client. At 
the macro level of analysis the unit of analysis could be the counselor, 
the office, the district or region, or the state VR agency; in each case 
several micro units of observation, i.e., clients, are combined or sub- 
sumed under the macro unit of analysis. 

While client-level microanalyses are essential for understanding how 
individual client outcomes are affected by client characteristics and the 
particular types of services received, the service system focus of macro- 
analysis is particularly useful for identifying practices which influence 
overall program performance. A macro level of analysis offers the fol- 
lowing advantages in investigating social service delivery system effective- 
ness: 

1. Certain planning and policy questions frequently asked are 
macro questions. The attempt to set standards to guide the performance 
of state VR agencies is a macro concern; the question is not whether 
individual clients are receiving quality services, etc., but whether the 
agency as a whole performs well. 

2. Given the short length of time clients are in most social service 
delivery systems, changes in the system over longer periods of time cannot 
be analyzed using client-level data. Even in a year-to-year analysis, there 
are still problems in linking pre- and post-data for a given client. With 
macro level data summarizing overall program performance, time series 
analysis becomes a possibility. 

3. The measurement of a macro -level phenomenon can be fundamentally 
different from that of a micro level phenomenon. The advantage of a 
macro analysis is that, the contextual effects of a particular program 
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with specific organizational structure, clients, and services can be assayed. 
Two clients in different programs may have the same characteristics and 
receive the same services; however, the organizational structure and the 
mix of other services and clients might result in a much different impact, 

4. Another advantage of a macro -level model is the ability of this 
level of analysis to examine the effect of environment. A strong emphasis 
is placed in macroanalysis on the role of environment as a limiting and an 
additive force. It is very difficult to measure the exact environment at 
the individual client level. Even if such measurement were possible (e.g., 
labor market conditions, local political attitudes about the disabled), 
there would be little variation from client to client, especially for those 
served by the same office or in the same local area. For example, the unem- 
ployment rate is only measurable at a macro level. If assigned to each 
individual in the area, then no variable would exist; there would be no 
variation, thereby preventing any analysis of the effect of this factor. 

5. In a micro analysis, individual differences and peculiarities 
come to the fore. In a macro analysis, these effects are wiped out. 
Neither is necessarily better. Sometimes the emphasis is on larger struc- 
tural effects, and for this, macro analysis is better. 1 

6. Micro data can always be aggregated, whereas disaggregation of 
macro data is often difficult and sometimes impossible. For this reason, 
micro analysis is often preferred to macro analysis. However, in social 
service delivery systems, a number of variables can be observed only at 

the macro level (e.g., agency procedures, staffing patterns). 

7. Certain concepts for understanding certain behaviors in a social 
service delivery system are macro concepts. For example, to understand 
client selection, the concept of population-at-risk is necessary, and this 
concept is a macro concept . 

Both macro and micro analysis are necessary for evaluation research 
for the data-based decision support system. For VR evaluation, micro 
analysis refers to client-level analysis, although some macro data could 
be included in that analysis (e.g., attaching counselor characteristics 



One circumstance where microanalysis is preferred is in attempts to 
trace the effect of the particular mix of services on client outcomes. 
Macro models, cannot exhibit a level of detail sufficient to address this 
issue . 



12 



8 



to data on each client served by a particular counselor). Macroanalysis 
refers to agency-level analysis. The focus of the federal standards sys- 
tem is on state VR agency behavior and, for RSA or national reporting, 
the macro level may be more useful. On the other hand, microanalysis of 

subunits within the states and of counselor behavior will be particularly 

> 

useful to individual state VR agencies. 

Decision support is a term that covers the activity of using standards 
information and other program information to answer questions about the 
state's attainment in the provision of rehabilitation services. Through 
the creative use of this approach to employing program information, man- 
agers can work in the identification of practices and environmental con- 
ditions affecting attainment. 
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II. CLIENT-LEVEL MICRO ANALYSIS 

Client-level analyses can play an important supportive role in the 
evaluation's research effort to investigate problematic performance. 
Client-level analyses fulfill two functions: 

• checking the accuracy, reliability, and validity of the infor- 
mation used in the standards data elements and in the informa- 
tional data elements; and 

• following up the program managers' investigations and following 
up the results of the agency-level analysis, in order to explore 
the detailed interaction between client characteristics, service 
interventions, and observed client outcomes. 

Each of these two functions is discussed further below. 
CHECKING VALIDITY, RELIABILITY, AND ACCURACY 

Description of Approach 

The careful work in developing appropriate standards and data elements 
will be ineffectual if the data used to report on and analyze attainment on 
the standards are suspect. To ensure that the data adequately support the 
desired analyses, we must be concerned with several, types of data quality. 

A very basic issue is that of validity : do the data measure the concept 
intended? If, for example, we rely on client reports to counselors of earn- 
ings at referral in a data element measuring change in economic independence, 
do these reports truly reflect the client's pre-VR earnings capability? 

Reliability , or the issue of consistency of data under repeated measure- 
ments, is a second concern, When looking at the same case file, and asked 
to determine whether improvement in gainful activity for a non-competitive 
closure has occurred, would two independent reviewers record the same answer? 

Accuracy in recording of data is a final concern. Do checks of the 
case file, agency vouchers or other independent sources show inaccuracy, per- 
haps through carelessness, in the recording of data items? Have out-of- 
range codes been recorded or keypunched? 
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In the pretest of the standards system in six state VR agencies, several 
issues of validity, reliability, and accuracy in the use of the new data col- 
lection instruments were examined. The results of this analysis are given 
in the Report on the Pretest of the Revised Vocational Rehabilitation Pro - 
gram Standards , Volume I, November 24, 1980. As the standards system begins 
to operate, the need for analyzing validity, reliability, or accuracy on any 
of the data elements may arise. 

Addressing each of these data quality issues calls for several different 
approaches. In terms of validity, the very process of selecting the data 
elements and data sources for the standards has been carefully designed 
to guarantee, to the extent possible, the validity of each measure. BPA's 
recommendations for data elements have been reviewed by RSA and state agency 
staff to identify data elements based on the best available evidence of 
their validity. Thus, the development of the standards and data elements 
has already benefited from existing work identifying validity problems with 
the measures. Nevertheless, periodic in-depth studies of the validity of 
key data elements may be required. BPA recommends that validity be re- 
examined at three to five year intervals. Depending upon the findings on 
validity problems with a given data element, they should be repeated more 
or less frequently as appropriate. Data items with no problems would not 
need to be studied as frequently; those with validity problems would be anal- 
yzed more regularly until resolution occurs. We recommend the validity stud- 
ies be done by RSA, perhaps through contracted studies, using a national 
sample or samples from selected states . 

BPA recommends that procedures for examining reliability also take the 
form of periodic studies to examine inter-rater reliability on data items 
requiring reviewer judgment and to analyze consistency in data definitions 
across states. Where results of these procedures identify reliability 
problems, they will be converted immediately to revised data elements or 
instructions for data collection to resolve the reliability problems. These 
reliability studies should also be* done by RSA, using a national or selected 
state samples, and should examine all items requiring reviewer judgment 
(such as improvement in gainful activity) and definitional issues (such 
as competitive versus non-competitive goal) . 
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The recommended procedure for examination of accuracy of data record- 
ing is the use of computer and, where appropriate, manual editing checks 
of recorded data to eliminate and correct out-of -range or "wild 11 codes, 
check for missing data, check for inconsistencies across data elements 
(e.g., a client who is listed as not having a job at closure, but who is 
listed as having retained a job at follow-up). These computer and manual 
edits for out-of -range codes, missing data, and consistency of recorded 
items will be routine procedures occurring annually with each data proces- 
sing cycle. 

Recommended Procedural Standard 9 is directly concerned with R-300 
validity and reliability, and somewhat with the validity and reliability 
of other reporting systems used by RSA. The main procedure to be used to 
measure performance on this standard is the case review process, using 
case folder information. Also suggested is occasional outside verification 
of certain data items by contact with employers, agencies handling public 
assistance, and so forth. 

Case Example: Checking the Accuracy and Validity of. Earnings at Referral 

In order to illustrate the kinds of data collection and analysis needed 
to check the validity, reliability, and accuracy of the standards data ele- 
ments, an example has been drawn from a BPA study for the Assistant Secretary 
for Planning and Evaluation (ASPE) and the Audit Agency of DHHS (Shea, et al . , 
1978). The example below concerns checking the earnings at referral data. 

To verify the accuracy and validity of the R-300 earnings data in that 
study, two basic procedures were used. First, in personal interviews, 
clients were asked to report on their earnings and employment situation 
for the periods in question. In addition to personal interviews, a second 
verification of validity was done using Social Security Administration 
Summary of Earnings records. Annual earnings data from the SSA were available 
for comparison with R-300 reports of earnings. Thus, comparisons among the 
R-300 data, client interview reports, and Social Security reports formed 
the basis for verifying the R-300 earnings data. 

Table 1 compares the weekly earnings at referral as reported on 
the R-300 with the weekly earnings reported by the client in the interview. 
The data show that in 79.6% of the cases, the R-300 figure was the same 
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as the client's reported figure. Nearly all of these matches were cases 
in which both data sources reported that the client had no earnings the 
week prior to referral. Thus, there was a relatively high degree of con- 
sistency between the R-300 and the client interview. 

Most of the cases where the R-300 was not equal to the client 
interview occurred when one of the sources reported no earnings and the 
other one. did (13. 8%), especially when the R-300 reported no earnings, 
but the client reported some. Thus, earnings the week of referral 
reported by clients in follow-up interviews indicates more earnings than 
does the R-300. A client may indeed have difficulty recalling, months 
later, the precise level of earnings during a particular week if he or 
she had earnings. However, since most clients are listed by the R-300 
as unemployed and thus presumably with zero earnings, this recall error 
should not be widely prevalent. A client may err in stating the amount 
of earnings but is less likely to err in stating whether there were any 
earnings. 

Table 1 

Weekly Earnings at Referral : 
R-300 versus Client Interview 
(N=1009, all cases) 



Proportion of clients for 


whom : 




R-300 more than interview 




. . 1.4% 






. . 1.2 






, 0.3 




interview reported no earnings. . 


. . 3.7 


R-300 equaled interview: 




. . 0.6 




. . 79.0 


R-300 less than interview 




. . 0.6 






. . 0.7 






2.2 






, 10.1 






100.0% 



This is calculated as: Client Interview Report - R-300 Report 

R-300 Report 
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To check the validity, we compared the R-300 weekly earnings 
at referral (adjusted to an annual figure) with the annual earnings 
the year prior to referral reported by the client in the interview, and 
then with the Social Security Administration earnings data. The same 
comparisons were done for the client interview reports of weekly earnings 
at referral to provide additional data on validity. 

The client interview question on earnings the year prior to referral 
recorded these data in 16 ordinal groups from "No earnings," "Less than 
$2,000," "$2,000 to $2,999," to "$25,000 and over." Table 2 provides 
a comparison of the extent to which the R-300 weekly earnings figures, 
extrapolated to annual figures, match the yearly earnings reported by 
clients in the interview. The table shows that only 44.7% did "match," 
with 31.7% being cases where the client had no reported earnings . 
Another 27.4% of the clients had no earnings the week prior to referral 
according to the R-300, but on the interview reported annual earnings of 
$1 - 1,999. While the R-300 weekly figure reports that 85.6% of the 
clients had no earnings the week prior to referral, only 33.1% of the 
same clients reported no earnings for the year prior to referral in 
their interviews. Thus, it seems that for a large proportion of the 
clients, use of annualized weekly earnings at referral reported on the 
R-300 will not reflect their earnings for the year prior to referral. 
On the other hand, since 66.1% of clients report earnings of less than 
$2,000 the year prior, the data are still consistent with the basic 
conclusion that most clients had minimal earnings expectations before 
receiving rehabilitation services. 

A comparison of the annualized R-300 weekly referral earnings figure 
with the Social Security Administration (SSA) earnings for the previous 
year substantiates the above conclusion about validity. Table 3 
reveals that in 39.9% of the cases both the SSA and the R-300 reported 
no earnings, but in 48.5% of the cases the R-300 reported no earnings 
when the SSA did. This seems to indicate that clients are frequently 
unemployed the week prior to referral yet have had earnings in the 
previous year. 
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Table 2 

Weekly Versus Annual Earnings at Referral: 
R-300 (Annualized)* versus Client Interview 
[N=722, all cases) 



Proportion of Clients for Whom : 

R-300 more than interview a 

R-300 some earnings, interview -- no earnings 1.4% 

R-300 and interview both project earnings 

R-300 some earnings, interview $1-1,999 5.6 

R-300 some earnings, interview more than $2,000 7.4 

Subtotal R-300, some earnings M.4% 

R-300 and interview both project no earnings 31.7 

R-300 less than interview 

R-300 no earnings, interview $1 - 1,999. 27.4 

R-300 no earnings, interview more than $2,000 26.5 

Subtotal interview, some earnings 66.9 

100.0% 

^his is calculated as (R300 Report) x (52) 

Table 3 

Weekly Versus Annual Earnings at Referral: 
R-300 (Annualized) versus SSA 
(N=922, all cases) 



Proportion of Clients for Whom : 

R-300 more than SSA: by 25% a or greater . 4.7% 

by 10-25% 0.3 

by less than 10% 1.8 

SSA reported no earnings. 2.4 

R-300 equaled SSA:" both reported some earnings . 0.0 

both reported no earnings 39.9 

R-300 less than SSA: by less than 10%. . . . 1.1 

by 10-25% 0.5 

by 25% or greater °- 8 

R-300 reported no earnings 48.5 

100.0% 



This is calculated as: (R-300 Report) (52) - SSA Report 

SSA Report 
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The validity of using the earnings the week prior to referral as a 
benchmark for pre-rehabilitation earnings was further checked by 
evaluating the use of client interview weekly earnings at referral. 
Perhaps the problem uncovered in validity stems from the inaccuracy of 
the R-300, rather than from any inherent non-validity of using weekly 
earnings at referral to estimate client income in the absence of 
rehabilitation. As Tables 4 and 5 indicate, however, whether R-300 
or interview reports of weekly earnings are used; the conclusion of non- 
validity remains the same. 

Table 4, which shows a comparison of annualized client interview 
weekly earnings at referral with the client interview reported earnings 
for the 12 months prior to referral, reveals that in only 42.1% of the 
cases did the two figures match, most of which (34%) were when both 
figures reported no earnings. Again, a large proportion of clients 
(43.5%) had reported no earnings the week before referral in their 
interview, yet reported some earnings in the 12-month period prior to 
referral. 

Table 5 presents a comparison of the annualized client interview 
weekly earnings at referral with the Social Security Administration 
annual earnings report. The table shows that all the cases in which the 
interview figure equaled the SSA figure were cases where both sources 
reported no earnings (39.2%). In 43.9% of the cases, the client reported 
no earnings the week prior to referral and the SSA reported some earnings 
during the year. 

The discrepancies between data sources for individual clients revealed 
by these analyses do not necessarily represent a problem for program manage- 
ment, however, unless individual data are the basis of program decisions. 
At the program level in the state and federal agencies, individual data are 
usually aggregated, with monitoring and evaluation primarily making use of- 
mean data, that is, data averaged across all clients. The Program Standards 
are set in terms of mean client data for earnings, including the cost 
benefit data elements. Given such uses, individual differences between 
data sources may prove to cancel out when aggregated. Table 6 presents 
an analysis of the magnitude of overall discrepancy by comparing 
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Table 4 

Weekly Versus Annual Earnings at Referral: 
Client Interview (Annualized) versus Client Interview 
(N=799, all cases) 



Proportion of Clients for Whom: 






, , 8.1% 
, , 34.0 




. , 43.5 




, . 0.0 


Weekly and annual both reported some earnings, but unequal. . 


. . 14.4 




100.0% 



Table 5 

Weekly Versus Annual Earnings at Referral : 
Interview (Extrapolated ) versus SSA 
(N=965, all cases) 



Proportion, of Clients for Whom : 

Interview less than SSA: by 25% a or greater 1.1* 

by 10-25% 1.5 

by less than 10% 1.9 

interview reported no earnings 43.9 

Interview equaled SSA: both reported some earnings 0.0 

both reported no earnings 39,2 

Interview greater than SSA: by less than 10% 1.4 

by 10-25%. . .......... 0.4 

by 25% or greater 7.6 

SSA reported no earnings . . .... 3.0 



100.0% 



^his is calculated as: (Client Interview Report) (52) - SSA Report 

SSA Report 
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Tabic 6 

Variations in Estimates of 
Clients 1 Earnings Potential Without Services , 
Using Aggregated Client Data From Different Data Sources 





Mean Client Earnings Data 


Data Source 


All 

All 

Clients, 
Closed and 
Open Cases 


All /""• 1 r~ nr\ 

All Liosea 
Cases 
08, 26, 
28, 30 


All PI APA/1 

Ail Lioseu 
Cases with 
Services 
26, 28, 30 


Successful 
Closures 
26 


SSA: Annual Earnings, 
Year Prior (clients 
with R-300 data 
available) 


$1561.93 
(922) 


$1647.80 
(379) 


$1667.39 
(248) 


$2073.77 
(162) 


R-300: Annualized 

Weekly Earnings at 
Referral 


$ 568.20 
(922) 


$ 632.38 
(379) 


$ 630.47 
(248) 


$ 889.12 
(162) 


R-300 as Percent of 
Social Security 


36 . 3% 
(922) 


38 . 3% 
(379) 


37 . 8% 
(248) 


42.8% 
(162) 


SSA: Annual Earnings, 
Year Prior (clients 
with interview data 
available) 


$1575.83 
(965) 


$1677.51 
(375) 


$1777.28 
(242) 


$2202.18 
(160) 


Interview : Annual ized 
Weekly Earnings at 
Referral 


$ 813.20 
(965) 


$1024.05 
(375) 


$1187.85 
(242) 


$1465.60 
(160) 


Interview as Percent of 
Social Security 


51.6% 
(965) 


61.0% 
(375) 


66 . 8% 
(242) 


66 . 5% 
(160) 
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aggregations of earnings across clients between data sources. The table 
shows that the inference of earnings without rehabilitation services 
which would be derived from an annualization of rehabilitants 1 weekly 
earnings at referral, as reported on the R-300, consistently is only 
36-43% of the inference which would be derived from year-prior Social 
Security records. If analysts use reports of weekly earnings prior 
to referral given by clients in interviews, the inference of earnings 
in the absence of rehabilitation services would be about 60% greater 
than using the R-300 but would still be only 51-67% of the inference 
based on Social Security records. While this higher estimate of earnings 
at referral by clients may simply reflect poor memory, the relatively 
greater consistency of client reports over Social Security data should 
provide some confidence in the interview data. 

All analyses are thus consistent in their findings. Many clients 
had no earnings the week prior to referral, yet were employed some time 
in the previous year. Earnings the week of referral systematically 
understates the pre-rehabilitation earnings capacity of VR clients. 
Finally, the magnitude of understatement is not trivial. 

EXPLANATORY CLIENT- LEVEL ANALYSES: FOLLOWING UP THE RESULTS OF OTHER 
ANALYSES 

Description of Approach 

As was noted in the discussion of investigating problematic attainment 
in the previous chapter, program managers may come to a point in tracing 
out possible problems where client-level analyses are required. For example, 
if the explanation for problematic attainment for cost per 26 closure was that 
not enough clients were being closed successfully (but enough clients were 
being closed) , an analysis examining what led to the lower success rate 
would be needed. If other state agencies were having this same problem, 
then an agency-level analysis would be called for. However, if this agency 
were the only agency having such a problem, or if the program manager had 
a specific client-level hypothesis, or if the agency-level analysis sug- 
gested a client-level explanation, then a client-level analysis would be 
required. 
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The structure of this client-level analysis is very simple. The 
analysis attempts to explain a dependent variable, such as successful 
closure, by a set of independent variables. These independent variables 
should include controlling variables, such as demographics, but they most 
importantly must include intervention variables, variables that RSA and 
the state VR agency can manipulate. By including the intervention variables, 
the explanations gotten from the analysis can easily be turned into cor- 
rective actions. 

Explanatory client-level analyses have- usually taken one of three 
forms: prediction of outcome, correlates of success, or production func- 
tion/cost effectiveness analysis. In studies of prediction of outcome, 
the attempt is to see whether the outcome of the rehabilitation process 
for an individual client can be predicted at referral or at intake, based 
on some characteristics of clients (see Worrall for a review of these 
studies). (Of course, in acidi ion to statistical prediction of outcome 
there is also clinical prediction, which is also implicit in the determin- 
ation of eligibility by the counselor.) In studies of correlates of suc- 
cess the attempt is to see what client and other characteristics are asso- 
ciated with success versus non-success, using data collected at any time 
during the process, including data collected at closure (also see Worrall). 
The production function/cost effectiveness anaLysis form is from economics, 
and has more recently seen use in research on VR (see Dodson, Skaburskis 
and Coll ignon for one of the few examples). (A production function is the 
relationship between quantities of. inputs and the maximum amount of output 
that can be produced.) The first two forms -- prediction of outcome and 
correlates of success — have been and are now the predominant ones in the 
field. However, the production function/cost effectiveness analysis form 
holds the roost promise for use in the supportive evaluation system. In the 
production function/cost effectiveness analysis form, the actual services 
received and the dollar amount of services are included in the analysis. 
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A Proposed Micro Modei 

A proper micro model using the statistical approach would specify 
the aspects of five components of the VR service delivery system: environ - 
ment , service agency , clients , services , and process results . The model 
is shown in Figure 1. Below, each of these components is further dis- 
cussed. 

The environment component includes the more general environment, 
which consists of factors such as region, urbanization, industrialization, and 
ethnicity and racial composition. A very large number of possible mea- 
sures of this general environment exists. However, in Dodson (1978) a 
very large number of measures of the environment were analyzed, and eight 
measures stood out: South vs. non-South, divorce rate, intranstate 
migration, unemployment rate, % urban, mortality rate, value added by 
manufacturing, and number of financial institutions per 100,000 population. 
In addition to the general environment, there are several other dimensions 
of the environment that are especially important for the VR service 
delivery system. One is the job market, the existence of jobs that can 
be held by the disabled. Another is public and employer attitudes toward 
the disabled, given the problems of stigma. A third is the medical infra- 
structure. A fourth is the physical environment, including the existence 
of cut curbs, ramps, barrier-free buildings, and suitable transportation 
facilities. A fifth is the availability of other private and public pro- 
grams. A sixth is the network of referral sources to channel clients 
to VR. Measures of the specific environment are not readily available, 
except for referral source for individual clients. 

The sorvice agency here includes everything but direct services. 
Included are the administrative structure, the program structure, method 
of service delivery, personnel, expenditures, non-direct services, caseload, 
client selection, program effort, state financial effort, and growth. 
Non-direct services are those services not given directly to the client, 
but instead are given in support of direct services. For example, train- 
ing is a direct service; outreach and job development are non-direct 
services. Program effort refers to the effort made by the agency to serve 
those who need services. 
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Figure 1 
Proposed Micro Model 
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The client component includes: demographic characteristics, nature 
and extent of problems requiring services, personal characteristics 
related to problems and services, personal preferences related to prob- 
lems and services, family situation, economic situation, and overall 
client difficulty. Demographic characteristics include, among other 
things, sex, age, race, education, martial status, and location (e.g., 
rural versus urban, shown to be important for vocational rehabilitation 
in Goodyear, Bitter, and Micek) . Nature and extent of problems requir- 
ing services refers to the primary problem of disability. Disability 
includes type of disability, existence of other secondary disabilities, 
severity of disability, and functional capability (see Haber for this 
last concept). Personal characteristics related to problems and ser- 
vices include personal skills (intelligence, aptitudes, training, exper- 
ience; see Tseng, 1972, for some measures), motivation, flexibility and 
adaptability, and perception of problems (see Walls and Miller for 
measurement of perception of disability; see Tseng, 1970, on locus of 
control, a similar concept). Personal preferences related to problems 
and services for vocational rehabilitation include preferences for voca- 
tion, for location of residence, for desired income versus desired leisure, 
and for lifestyle. The family situation refers to the support - emotional, 
personal, and financial - provided by the family. The economic situation 
includes labor force participation, earnings, income, sources of support, 
number of dependents, and occupation and industry. Client difficulty is 
the difficulty of successfully serving a client, of attaining a certain 
process result. 

In VR, client difficulty is also called case difficulty. One approach 
to define difficult clients is to use empirical results from the litera- 
tures on prediction of outcome and correlates of success. Worrall surveys 
75 such studies and identifies several recurring predictors and correlates 
of outcome and success: age (younger), race (white), education (higher), 
and marital status (married). He also notes that these "vari:^ are 
also significant in studies of labor-force participation; and, 
employment is a common measure of rehabilitation program success, we 
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might expect to find them significant in rehabilitation outcome studies" 
(p. 110). Although client difficulty may be inferred from other charac- 
teristics (or it may not) or may be reflected in the outcome or result of 
the process (or it may not), client difficulty is not directly measured. 
The second approach, less used, is to directly measure client difficulty 
(see Westerheide, Lenhart, and Miller for one suggested measure; see 
Wright, Reagles, and Thomas for another) . However, this approach requires 
costly interviewing, testing, and observation. A third approach is to de- 
fine a priori cases that are difficult or easy. This a priori definition 
could be based on experience and a review of objectives. RSA takes this 
approach in a memorandum on "Services to Clients with Insubstantial Employ- 
ment Handicaps: Selected Disabling Conditions of Clients Rehabilitated in 
Fiscal Year 1973." RSA identifies several disability groups which it be- 
lieves have "slight or inconsequential disabilities which do not constitute 
substantial barriers to employment and, thus, are either ineligible for 
services or do not require VR intervention" (p. 1). These groups include: 
digestive system conditions (including dental conditions and hernia) , hear- 
ing impairments other than deafness, other mental illness (character, per- 
sonality, and behavior disorders other than psychosis, psychoneurosis , drug 
addiction, and alcoholism), mild mental retardation, and hay fever/asthma. 
Of the possible client variables, only demographic characteristics , disability 
(except functional capability), and economic situation are included in the 
R-300. Personal characteristics related to problems and services, personal 
preferences, and family situation (except monthly family income at referral) 
are not included. No direct measure of client difficulty is contained in the 
R-300, although one could be constructed. 

The services component includes the types and amounts of services 
offered. Moreover, the mix and interactions of services is also important 
(see Dodson, Skaburskis, and Collignon for one of the few analyses using mix 
and interactions of services) . For example, in VR, training without needed 
restoration services may not be vocationally fruitful. In addition to 
quantity of services, the quality of services is of consequence. Whether 
a technology exists for serving clients is even more important, determin- 
ing the appropriateness of services. Who provides (pays for) the service 
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and who produces (delivers) the service are also of significance. Voca- 
tional rehabilitation services offered by the agency include the formal 
purchased services plus the services of the counselor. 

Unfortunately, the VR R-300 reporting system includes information 
only on whether each of the types of formal services are provided, and 
a few cost categories. The services provided are noted as provided with 
cost, with no cost, or with partial cost to the agency. Cost categories 
include total costs, rehabilitation facilities costs, Social Security 
Trust Funds costs, and extended evaluation costs. That is, no cost by 
type of service is provided. No costs are recorded for services paid 
for by sources other than the VR agency. Moreover, services provided by . 
the family are not included, either as delivered or in costs. Participa- 
tion in certain special federal and state programs is recorded on the 
R-300, however, thereby allowing a glimpse at some of the other public 
agencies providing services. However, no information on counselor services 
is provided. And, of course, no quality measures are available on the R-300. 
For referral, the reason for nonacceptance and where the client is referred 
to if not accepted are recorded on the R-300. The appropriateness of 
services and the implied technology have not received very much discussion 
in VR. In addition to vocational services, the client may receive other 
services from public and private agencies. The other services listed on 
the R-300 include SSDI benefits (status at referral and at closure) and 
public assistance payments (amount at referral and at closure, amount of 
time on public assistance at referral, and type of public assistance at 
referral and at closure) — both maintenance services. 

Process results can be measured in many different ways and at many 
different points in the process . Measures include: 

1. acceptance or rejection for services — eligibility; 

2. completion or non-completion of the process; 

3. reliability of the plan for services r- whether planned 
services are delivered at all, and whether in the correct 
sequence, mix, and amount; 

4. return for services; 
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5. level of outcome; 

6. change in level of outcome; 

7. quality of outcome; 

8. persistence or permanence of outcome — at follow-up 
whether retention of benefits; 

9. level of public support still required; 

10. release from or continuance in an institution; 

11. administrative success — whether considered successful 
by service deliverer given agency rules and standards; 

12. client satisfaction with services; 

13. client involvement in planning and in service delivery; 

14. time in process; 

15. personal development; 

16. increased group awareness and group organizing. 

The application of these process results measures to the VR system 
follows. Eligiblity is given by closure status; 08s are ineligible. Com- 
pletion or non-completion of the process is given by the closure status, 
although the reason for closure would be more useful (but is not recorded 
on the final R-300, though on the form). The plan is not recorded on the 
R-300, and thus the reliability of the plan cannot be gauged. Return for 
services is recorded for clients before but not after; whether there has 
been a previous closure within three years, type of closure, and months 
since previous closure are all recorded. The outcome, measurement has 
several dimensions, corresponding to the objectives of the VR program. 
Employment is measured by closure status 26 and work status at closure, 
and weekly earnings at closure. Homemaking is indicated by closure status 
26 and work status at closure. No measure of self-care or of independent 
daily living is available on the R-300 (see Dinnerstein, Lowenthal, and 
Dexter for one such measure). Reduction in public assistance rolls is in- 
dicated by type and amount of public assistance at referral and at closure 
Change in level of outcome can be computed using a before and after ap- 
proach. For example, the change in weekly earnings from referral to clo- 
sure can be be computed. The question is whether the client gains (see 
Reagles, Wright, and Bulter for the concept of rehabilitation gain; see 
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Hawryluk for a criticism) and how much the client gains from the vocational 
rehabilitation process. Quality of outcome is not measured, along any of 
the types of outcomes. One type of measure for quality of employment is 
whether final occupation fits the occupational goal (see Shea, et_. al . , 
1978). Retention of benefits is not available on the R-300, but the 
follow-up instrument asks this question of a sample of closures. The 
level of public support still required is given by public assistance, 
SSDI, and SSI status at closure, and whether public support is needed 
later is also available from the follow-up survey, The institutional 
status of clients is not recorded. Administrative success is given by the 
26 closure. A client satisfaction measure is available from the closure 
survey. Client involvement in planning and in service delivery is covered 
by Procedural Standard 12 on the IWRP. Time in process is given in several 
categories — total months, plus months in certain statuses . Nothing about 
personal development or group awareness or group organizing is recorded on the R 

Case Example: Explanation of Client Outcomes 

The previous discussion of the proposed micro model has been largely 
theoretical. But, what will such an analysis look like in the supportive 
evaluation system? Below is an example of explaining various client out- 
comes,* many of which are incorporated into the standards data elements, 
This example illustrates the hypotheses examined, the measures used, the 
analytical and statistical techniques employed, and the types of conclu- 
sions that can be drawn from such an analysis. 



Also drawn from the Shea, et. al. (1978) study. 
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In attempting to determine what contributes to successful client out- 
comes and, thus, what program improvements will enhance VR's effectiveness, 
we have concentrated on several measures of client success. In addition to 
the traditional 26 closure measure, we have analyzed the more rigorous 
employment and economic outcome measures presented earlier: achievement of 
competitive employment at closure, minimum wages at closure, and retention 
of wages after closure. The selection of this set of variables permits a 
comparison between the program definition of success and other more specific 
outcome measures. We also look at client satisfaction with VR. This is of 
interest for two reasons. First, it enables us to address recent program 
(and standards) emphasis on client satisfaction. Second, since our findings 
suggest satisfaction measures have little relation to program outcomes, it 
will be instructive to see whether satisfaction is conditioned by factors 
different from those which condition other program outcomes. 

The final list of variables used for analysis thus includes the follow- 

1 

ing: A 

• The 26-closure: client was closed 26 vs. client was closed 
28 or 30. 

0 Competitive employment: client was closed into competitive 

employment vs. client was closed into non-competitive employment. 
The latter group includes those unemployed at closure. 

• Achievement of minimum wage: client earned the minimum wage at 
closure vs. client did not earn minimum wage at closure. The 
latter group includes those with no earnings at closure. 

• Wage retention: a comparison of wages at closure and at 
follow-up. Clients were considered a "success" if follow-up 
wages equalled or bettered those at closure. Clients were not 
considered a "success" if either: (a)— follow-up wages were less 
than closure wages; or (b) the client had zero earnings at closure. 

• VR satisfaction: client was either "very satisfied" or "somewhat 
satisfied" with VR services vs. client was "very dissatisfied," 

*Each variable is dichotomous, i.e., is measured in terms of "success" (1) 
or "failure" (0). 
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"somewhat dissatisfied, 11 or "neither satisfied nor dissatisfied 11 
with VR services. 

In an attempt to explain differences in client outcomes, we iQoked 
at a series of variables representing (1) client characterististics , (2) 
the client's service process experience, and (3) counselor characteristics 
and agency context. 

Client Characteristics . We included such characteristics as age, sex, 
and race, as well as education and previous work experience, which may be 
expected to have an observed effect on client outcomes: much previous 
research has documented that young, white males achieve greater success 
in the general labor market. Literature specific to VR has also documented 
age, sex, race, and employment potential characteristics as correlates of 
success. 

A client's specific disability, as well as the severity of the disability 
might also show relationships to our dependent variables. It may be that 
clients with certain disabilities face greater obstacles to employment. 
Further, severely disabled CSD) clients, or those clients with secondary 
disabilities, may require greater service investment and represent cases 
most difficult to rehabilitate. Thus, it is possible that such clients 
will fare worse than other clients in terms of VR outcomes. In this study, 
we were particularly interested, given the 1973 provisions for SD priority, 
to observe outcomes for the severely disabled in relation to those of non- 
SD clients. 

Finally, we included several broad indicators of the client's Employ- 
ment "potential," including the client's level of education and the client's 
work experience previous to becoming a VR client. In terms of work experi- 
ence, clients were classified into four groups: those with no work experi- 
ence; those with work experience, but who worked only before becoming 
disabled; those who worked after becoming disabled,* but who were not working 
at referral; and those working at referral. Depending on a client's age 
and/or case severity, lack of any work experience may serve to bias coun- 
selors against heavy investment in hira or her.-. And if nothing else, lack 
of work experience provides some indication of innate employment potential, 
again depending on age and case severity. 
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Client welfare status at referral may affect later vocational out- 
comes, and was also included as a variable in the analyses. Welfare 
recipients may represent cases more difficult to rehabilitate. In addition, 
such clients may experience economic disincentives which bias them against 
successful outcomes. 

Service Process Experience . Service process variables selected for 
inclusion in the analyses included compliance with IWRP requirements, 
which may facilitate clients' involvement in their own rehabilitation 
processes. This may, in turn, increase the probability that such clients 
will "do their share" toward their own rehabilitation and that the agency 
will carry through its responsibilities more completely, thereby increasing 
the probability for successful outcomes. 

Other process variables potentially important to an analysis of out- 
comes include various indicators of smooth service flow. For example, 
reliable delivery of services (i.e., completion of all services planned for 
the client) might facilitate favorable outcomes, since reliability implies 
service continuity and follow -through. Other indicators of service flow 
include elapsed time from referral to acceptance, and from referral to 
closure. Other things being equal, clients receiving timely service may 
have greater chances for success, since they have been spared potentially 
discouraging delays in service provision. On~the other hand; clients who 
have had more than one counselor may suffer from a lack of service continuity 
and thus may have decreased chances for success. Finally, clients whose 
cases were reviewed by supervisors may experience outcomes different from 
those of clients never reviewed by supervisors. Possibly, such clients may 
benefit from supervisor redirection of their individual rehabilitation 
program. 

Several service-related issues may also be relevant to client outcomes. 
We looked at whether the number of services planned for a client was asso- 
ciated with probability for success. Larger numbers of services planned may 
be associated with favorable outcomes, since more services imply heavier 
VR investment for such clients. Provision of various specific services may 
increase chances for favorable outcomes. In particular, we wished to know 
whether provision of employment -oriented services increases chances for 
success. 
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Counselor Characteristics and Agency Context . Several factors may 
affect counselors' abilities to perform their jobs and thereby influence 
client outcomes. Our analysis included measures of the counselors' educa- 
tion, receipt of in -house training, and experience as a VR counselor. With 
reference to education, we were interested in determining whether counselors 
with degrees in rehabilitation achieved greater success. Such a finding might 
be expected, since these counselors presumably are most knowledgeable in 
rehab issues and techniques. Provision of in-house training may also faci- 
litate client success; such training may augment counselors' expertise, if 
it helps counselors keep abreast of current developments aimed at improved * 
performance. Finally, more experienced counselors may have greater familiar- 
ity with the subtle nuances of effective counseling; thus, such counselors 
may be able to deal more effectively with clients of diverse circumstances. 
A counselor's perception of his or her role in the rehabilitation process 
might have an impact on client success. For example, counselors who rate 
vocational counseling as their first priority may achieve greater success 
with their clients. Alternatively, such counselors may experience varying 
degrees of success across outcome variables; for example, these counselors 
may place more clients in employment but fail to help their clienrs achieve 
reasonable earnings, if employment is the overriding concern. 

Indicators of counselor productivity were also expected to be relevant 
in an analysis of outcomes. Concern has been expressed xhat agency 
pressures for increased performance may hinder client success, if such 
pressures incline counselors toward hasty planning and quick closure, i.e., 
a focus on quantity rather than quality. Our analysis included measures 
of the counselor's number of 26 closures as an indicator of pressure to 
produce . 

Counselor caseload size may also be a variable, representing agency 
and counselor context, with impact on outcomes. While the obvious hypo- 
thesis would be that larger caseloads would hinder success, because the 
counselors with extremely large caseloads would be overburdened, there 
have been several studies in VR showing the opposite relationship, suggest- 
ing that the more able counselors feel capable of taking on larger respon- 
sibility and can serve more cases. 



ERIC So 



31 



The final counselor characteristic included in our analysis was a 
measure of counselor autonomy. Counselors were asked how often they checked 
with their supervisors regarding service decisions. !, Autonomous" counselors 
are those responding that they "didn't usually 11 or "only occasionally" 
checked with their supervisors; "non- autonomous" counselors are those 
responding that they "usually" checked or that they "are required" to 
check with their supervisors. It may be that autonomous counselors achieve 
less success if they are freer to "cut comers" in service and closure 
decisions. Alternatively, autonomous counselors may be those most able to 
manage their cases without assistance from superiors. If so, autonomous 
counselors may achieve greater success. 



The first issue addressed in the analysis concerned the relative influ- 
ence on outcomes of these three general sets of variables — client characteris- 
tics, service process, and counselor characteristics. In this case, we wished 
to know whether client characteristics, service process experience, and 
counselor characteristics/agency contexts were relatively equal in their 
effect on outcome, or whether one set of factors dominated. Determining 
relative influence allows us to identify general targets at which to aim 
policies designed to improve VR's performance. For example, a finding that 
elements of the service process exerted little influence in relation to 
client characteristics would suggest two conclusions. First, we might con- 
clude that what a client brings to VR in terms of potential and employability 
most strongly conditions the client's probability for success, and that efforts 
to change such things as time in process, service reliability, etc., would 
yield little improvement in outcomes. Second, such a finding might justify 
a recommendation that VR reassess its service strategies and levels of 
effort for different client groups to compensate for the inherent disad- 
vantages of some types of clients. Alternatively, such a reassessment 
might show that certain client groups systematically receive inferior 
treatment; or, perhaps, it might be found that these groups more appro- 
priately belong in public service programs other than VR. 
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We assessed relative impact through use of variance partitioning 
techniques. Our findings show that, in fact, client characteristics 
tended to be the most potent predictors of overall outcomes (see Table 
7). The percentage of variance uniquely explained by client characteris- 
tics equalled or exceeded that of the other two blocks for all dependent 
variables except closure status. 



Table 7 

Variance Partitioning for Regressions on Outcome 
of Accepted Clients 



Dependent 




Explained Variance 


a 


-1 


Variable 




Unique 


i 




Total 


Client 


Process 


Counselor 


Shared ! 


Closed 26 


.62787 


.10250 


.21884 


.05592 


.25060 J 


Competitive Employment 


.74753 


.26324 


.19536 


.04628 


.24265 | 

— i 


Minimum Wage 


.60525 


.28901 


.11154 


.07979 


.12491 


Wage Retention 


.64329 


.24179 


.12830 


.03676 


.23644 


VR Satisfaction 


.50412 


.11486 


.09572 


.11573 


.11781 



a Unadjusted R 



Employment and earnings outcomes were particularly influenced by client 
characteristics, which accounted for one-quarter of the total variance on 
the competitive employment, minimum wage, and earnings retention variables . 
Nonetheless, service process variables did exert considerable influence on 
employment and earnings. Achievement of competitive employment, in particu- 
lar, appeared to be strongly influenced by service strategy and the individual's 
experience while a VR client. In contrast, counselor characteristics had 
little relative impact on employment and earnings: only on the minimum wage 
variable did counselor characteristics uniquely account for as much as 8% of 
total variance. 

The probability of a "successful" closure was most influenced by service 
process variables, which uniquely accounted for 22% of total variance. How- 
ever, the individual client's characteristics did impact somewhat on closure 
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status. Here again, counselor characteristics exerted little influence. 
One explanation for the stronger impact of process variables on closure 
status may be that the 26 closure is "easier 11 to achieve than are competitive 
employment and minimum wage earnings. Thus, client characteristics may 
present less of an obstacle to the 26 closure. Such an explanation is 
sensible, inasmuch as the 26 closure includes homemakers: a non-working, 
non-wage-earning group. Put another way, the broadness of the 26 closure 

m 

permits recognition of the widely varying potentials of clients and that 
valid rehabilitation, under current regulations and legislation; can mean 
very different things. Employment outcomes represent a narrower definition 
of rehabilitation, and we would expect client characteristics associated with 
labor market receptivity to have greater impact. 

Satisfaction with VR showed very different relationships to the indepen- 
dent variable blocks. Each of the blocks uniquely accounted for approximately 
the same proportion of variance (10%). This is interesting because, first, in 
this sole case counselor characteristics exhibited an explanatory role equally 
as powerful as found for client and process variables. This suggests that 
satisfaction partially reflects the client's feeling toward his or her counselor. 
Second, process variables have the least impact on satisfaction (albeit by a 
small margin) , justifying the conclusion that satisfaction is not primarily 
a function of effective planning and/or service provision. 

In sum, the above analysis suggests the following general conclusions: 

(1) The client f s innate potential, both for rehabilitation and 



certain client groups. Such efforts might include rigorous 
examination of the special problems experienced by these 
groups, both as clients of VR and as job-seekers. 

(2) Nonetheless, process variables did impact on employment 

and earnings. Thus, we can be reasonably confident of the 
efficacy of program elements. In the next stage of 
analysis we will see which program elements bear the 
strongest relation to outcomes. 



employment, most strongly conditioned his or her probability for 
achieving competitive employment and reasonable wages. This 
suggests that concerted efforts may be in order on behalf of 
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(3) As expected, probability of 26 closures and client satis- 
faction showed somewhat unique relationships to the inde- 
pendent variables. The less rigorous requirements of the 
26 closure, in relation to employment and earning criteria, 
may account for the different patterns observed for closure 
status. In contrast, client satisfaction is much more 
influenced by counselor characteristics, while program 
elements (theoretically the basis for subjective extermi- 
nation of "satisfaction") exert the least influence. 

While variance partitioning allows assessment of relative impact of 
variable blocks, it does not provide information on the effect of each 
variable; this requires a detailed analysis of coefficients and significance 
levels. In addition to identifying the most important independent variables, 
i.e., factors affecting outcome, such an analysis also allows further compari- 
son of the dependent variables or outcome measures themselves. Table '8 
presents the analysis of factors affecting client outcome. 

Prior to detailed analysis of the regressions, to aid in understanding 
the explanatory power of our regressions for our outcome measures, classi- 
fication tables are presented. A predicted value for each case is calcu- 
lated using the regression coefficients and the values of the independent 
variables for that case. Then this predicted variable is dichotomized, 
"no" if the predicted probability is under .5, and "yes 11 if over .5. 
Table 9 indicates that our regressions do a fairly good job of predic- 
tion: more than 80% of our clients were correctly classified on each 
dependent variable (that is, are no-no or yes-yes), and in most cases the 
proportion is closer to 90%. Further, the proportions of prediction errors 
generally are evenly distributed. For example, 4.5% of our clients were 
closed 26 when our equation predicted they would be closed 28 or 30; 
likewise, the regression equation predicted 4.4% would be closed 28 or 30 
when in fact these clients were closed 26. Only for the minimum wage and 
VR satisfaction measures does this pattern not hold. Thus, in general, our 
model provided accurate predictions of ultimate client outcomes. 

As expected, many client characteristics showed strong, consistent 
relationships to employment and earnings outcomes. This was primarily true 
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Table 8 

Regression for Explanation of Client Outcomes 
(Closed Clients, 26, 28, 30) 



INDEPl.NDENT 



DEPENDENT 



Closed 26 



Competitively 
Employed 



Earned Minimum 
Wage or Better 



Wage 
Retention 



VK 

Satisfaction 



Client Characteristics 

Arc: 

16-J4 

25-44 a 

45* 

Male 

White 

Education: 

0-8 years 

9.11 years 

High school a 

More than high school 

Severely disabled 

Has second disability 

Primary Disability: 
Visual 

Aural 

Mental illness 
Drug and behavioral 
Orthopedic, amputee 
Mentally retarded 
Other 

Receiving wplfsre & itftml 
Work experience pre-VR: 



(.10) 



-.01 
(.10) 

-.03 
(.08) 

.22 * 
(.09) 



-.15 
(.10) 

.09 
(.10) 



.16 
(.14) 

-.05 
(.08) 

.08 
(.10) 



.20 
(.21) 

.07 
(.19) 
-.05 
(.11) 

.09 
(.12) 



.19 
(.17) 

.07 
(.11) 



5tM " 
(ilO) 



-.20 • 
(.08) 



-.20 • 
(.09) 

.11 
(.07) 

.20 * 
(.08) 



-.32 • 
(.09) 

-.03 
(.08) 



#19 
(.12) 

-.09 
(.07) 

-.03 
(.08) 



.06 
(.18) 

.01 
(.16) 

.03 
(.10) 

.19 
(.10) 



.10 
(.14) 

.07 
(.09) 

»»14 
(*08) 



-.38 
(.12) 



-.48 * 
(.13) 

.21 • 
(.10) 

.24 • 
(.11) 



-.04 
(.13) 

.14 
(.12) 



.05 
(.17) 

-.10 
(.10) 

.12 
(.12) 



.46 
(.26) 

-.07 
(.24) 

.01 
(.14) 

-.17 
(.IS) 



-.33 
(.21) 

-.14 
(.13) 

-.23 
(.12) 



-.20/ 
(.10) 



-.16 
(.10) 

.15 
(.08) 

.12 
(.09) 



-.10 
(.10) 

-.02 
(.10) 



-.14 
(.14) 

-.01 
(.08) 

-.07 
(.10) 



-.03 
(.21) 
-.08 
(.19) 
.14 
(.11) 
-.00 
(.12) 



.27 
(.17) 

-.13 
(.11) 

-.20 • 
(.10) 



-.IJ 
(.11) 



.10 

(.11) 

-.12 
(.09) 

.02 
(.10) 



.03 
(.11) 
-.03 
(.11) 



.13 
(.15) 

.12 
(.09) 

-.14 
(.11) 



-.06 
(.23) 

.22 
(.21) 

-.09 
(.13) 

.05 
(.13) 



.OS 
(.19) 
..20 
(.12) 

-.04 
(.11) 



Never worked 


.07 


-.26 ** 


-.39 ** 


-.48 ** 


.09 




(.12) 


(.10) 


(.IS) 


(.12) 


(.13) 


Worked, only pre-disability 


-.17 ♦ 


-.49 ** 


-.6S ** 


-.58 ** 


-.01 


(.12) 


(.10) 


(.15) 


(.12) 


(.13) 


Worked after disability, 


-.07 


-.18 ** 


-.40 


-.30 ** 


-.10 


but not at referral 


(.10) 


(.09) 


(-13) 


(.10) 


(.11) 


Working at referral 3 













First number is the unstandardized regression coefficient 
variable. Second number (in parentheses) is the standard 



representing effect of independent on dependent 
error of the coefficient. 



Categories for which no dummy variable was created ("left out" category). 
^Continuous variables; coefficients give range of effect. 
Significance levels for coefficients noted as follows: **■ 



.000 to .050; **.051 to .100; ♦=.101 to .200. 
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Table 8 (continued) 



INDEPENDENT 



DHriiNDUNT 



Closed 20 Competitively named Minimum 
Employed Wage or Rctter 



Wage 
Retention 



Vtt 

Satisfaction 



Process Variables 
Has IKRP 

Plan reliability 

Time from referral 
to acceptance b 

Time from referral 
to closure** 

Mad more than one counselor 

Supervisor reviewed case 

Number of services planned** 

Services delivered: 
Prosthetics 

Diagnostics and 
evaluation 

Education 

Vocational training 
On-the-job training 
Tools 

Job placement 
Transportation 
Maintenance 
Family services 
Follow-up services 
Other 



+ .26 


.30 ** 


.08 


(.09) 


(.07J 


f 111 


.19 * 


-.11 ♦ 


.07 


(.10) 


(.09) 


f 121 


- 00 -.46 


.00,. 52 


.00,. 22 


(.0004) 


(.0003) 


(.0005) 


.00,. 25 


.00,-07 


-.00, -.57 


(.0001) 


(.0001) 


(.0001) 


i n 


.04 


.14 ♦ 




(.07) 


(.11) 


-.01 


-.06 


-.36 ** 


(.08) 


(.07) 


(.10) 


.03 .27 


-.07,-. 58" 


.03,. 23 


(.03) 


(.02) 


(.03) 


.27 


.44 


.12 


(.11) 


(.09) 




.03 


.15 ** 


.10 


(.08) 


(.07) 


(.10) 


.21 


.26 ♦ 


.59 ** 


(.20) 


(.17) 


(.25) 


.21 * 


.45 ** 


.18 


(.12) 


(.10) 


(.15) 


.19 


-.03 


.03 


(.19) 


(.17) 


(.24) 


-.09 


.20 * 


-.16 


(.13) 


(.11) 


(• JOJ 


.23 * 


.27 ** 


.13 


(.12) 


(.11) 


(.15) 


.09 


-.03 


-.03 


(.14) 


(.12) 


(.17) 


-.15 


-.02 


-.11 


(.14) 


(.12) 


(.17) 


-.50 ** 


-.09 


-.10 


(.25) 


(.21) 


(.31) 


.13 


.04 


.12 


(.16) 


(.13) 


(.20) 


.06 


.05 


-.07 


(.10) 


(.08) 


(.12) 



.12 ♦ 
(.09) 

-.16 ♦ 
(.10) 

.00, .24 
(.0004) 

-.00, -.43 
(.0001) 

-.01 
(.09) 

-.16 • 
(.08) 



.15 ♦ 
(.09) 

.16 ♦ 
(.11) 

-.00, -.54 
(.0004) 

.00,. 01 
(.0001) 

-.01 
(.09) 

-.17 • 
(.09) 



-.02, -.21 


.03, .23 


(.03) 


(.03) 


.24 ** 


.04 


(.11) 


(.12) 


.10 


.03 


(.08) 


(.09) 


.46 ** 


.11 


(.20) 


(.22) 


.25 ** 


-.05 


(.12) 


(.13) 


.27 ♦ 


-.33 


(.20) 


(.21) 


-.19 ♦ 


.10 


(.13) 


(.14) 


.29 ** 


-.18 ♦ 


(.12) 


(.14) 


.05 


.19 


(.14) 


(.15) 


.09 


.00 


(.14) 


(.15) 


.24 


-.20 


(.25) 


(.27) 


-.01 


.07 


(.16) 


(.17) 


.13 ♦ 


.01 


(.10) 


(.11) 



T First number is the unstandardited regression coetticient nr»»«i "„,-il«t 
var!ab"r Second number (in parentheses) is the standard error of the coefficient. 
Categories for which no dummy variable was created ("left out" category). 
b Continous variables; coefficients give range of effect. 

Significance levels for coefficients noted as follows: -.000 to .050; -.051 to .100; .-.101 to .200. 
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Table 8 (continued) 



INDCriNUDNT 


DEPENDENT 












Closed 26 


Compet it jvely 


Earned Minimum 


Wage 


VR 






Employed 


Wage or Better 


Retention 


Satisfaction 


Counselor Characteristics 












Education: 












Less than BA 


+ .0U 


-. 12 


-.31 ♦ 


-.16 


-.02 




CIS') 


(.16) 


(.23) 


(.19) 


(.^1) 


BA 


.06 


-.11 


-.27 ♦ 


-.05 


.22 ♦ 




(.13) 


(.11) 


(.17) 


(.13) 


(.15) 


Some graduate 


.06 


-.01 


-.15 


-.08 


.09 




(.13) 


(.11) 


(.16) 


(.13) 


(.15) 


Graduate: rehabilitation 


— 


— 


-- 


-- 


— 


counsel ing a 












Graduate: psychology, socio— 


. 17 


— • Lt * 


— • J J ♦ 


-.19 




l°g>*» special education 


(.19) 


(.16) 


(.23) 


(.19) 


(.20) 


Graduate: counseling and 


. 05 


" . 1 £ 


. - • 1 ' 


nn 
-. UU 


-.11 


guidance 


(!l5) 


(.11) 


(.17) 


(.13) 


f .151 


Graduate : other 


.04 


. 1 3 




nA 


i a 




(.16) 


(.14) 


(.20) 


(.16) 


f.181 


Received training in : 














. 10 


m 


• UD 


nn 
.09 


• 15 ♦ 




(.08) 


(.07) 


(.10) 


(.08) 


f.091 


Job market requirements 


.01 


. 02 


-.03 


-. 14 * 


ni 




(.08) 


(.07) 


(.10) 


(.08) 


f .091 




-. 09 


- . 07 


- • uo 


-.06 


-.10 




('.10) 


(.08) 


(.12) 


(.10) 


(.11) 


Counselor experience: 












l JidJl £ /Cars 


i n 

♦ 1 U 


.19 ** 


.19 ♦ 


-.04 


.17 ♦ 




(.11) 


(.101 


f . 141 


f 111 


f 121 




_ 1 A * 
— . J** ▼ 


-. u / 


.17 ♦ 


-.01 


-.02 




(.09) 


(.08) 


(.11) 


(.09) 


(.10) 


Over 5 years 3 




— 


— 


— 




Perception of role: 












Diagnostics: testing 


.02 


.04 


.17 ♦ 


.04 


-.03 


and evaluation 


(.10) 


(.09) 


(.12) 


(.10) 


(.11) 


Vocational counseling 3 


— 
— 




-- 




— 


— 
-- 


-- 


General counseling 


-.01 


.07 


-.03 


-.04 


-.10 




(.09) 


(.08) 


(.11) 


(.09) 


(.10) 


Referral 


-.29 


.08 


-.34 


-.11 


-.12 




(.23) 


(.20) 


(.28) 


(.23) 


(.25) 


Job placement 


.14 


.08 


.33 * 


.02 


-.09 




(.15) 


(.12) 


(.18) 


(.15) 


(.16) 


b 

Number of 26s 


.00.. 21 


-.00.-. 17 


.00. .23 


-.00,-. 17 


-.00.-. 11 




I. VVt) 


f nm ^ 


f nn^ 


l.UQJj 


t nn ^\ 
(.UU-J 




-.00 -.07 


nn t?a 


nn a i 

. UU | .4 1 


nn io 
• UU . . 


nn C£« 
. UU . • 3o 




(.001)' 


(.001) 


(.001) 


(.001) 


(.001) 


Autonomy: does not usually. 


.14 


-.10 


-.20 ♦ 


-.07 


.15 


or only occasionally checks 


(.12) 


(.10) 


(.15) 


(.12) 


(.13) 


with <unAn'1 ?or 












1 4 
-• 14 


.41 


m 


-.15 


.21 


Significance (F) 


.000 


.000 


.000 


.000 


.001 


t correctly classified 


91.1 


90.7 


83.1 


86.4 


89.7 


R 2 


.63 


.75 


.61 


.64 


.50 


Adjusted R 2 


.44 


.62 


.41 


.47 


.26 


N 


I'M . 


170 


170 


170 


170 



First number is the unstandardized regression coefficient representing effect of independent on dependent 
variable. Second number (in parentheses) is the standard error of the coefficient. 

Categories for which no dummy variable was created ("left out" category). 

^Continous variables; coefficients give range of effect. 

Significance levels for coefficients noted as follows: ***.000 to .050; *.051 to .100; ->*.101 to .200. 
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Table 9 

Classification Tables for Outcomes Measures: 
Statuses 26, 28 and 30 



Prediction: 26 Closure 


Client Closed 26? 
No Yes 


Total 


No 


34 a b 5 
(27.9%) (4.5%) 


40 

(32.4%) 


Yes 


5 77 
. (4.4%) (63.2%) 


83 

(67.6%) 


Total 


40° 83 
(32.3%) (67.7%) 


122 


Prediction: Competitive 
Employment 


Client Competitively 
Employed? 

No Yes 


Total 


No 


41 5 
(38.0%) (5.0%) 


46 

(43.1%) 


Yes 


5 57 
(4.3%) (52.7%) 


61 

(56.9%) 


Total 


46 62 
(42.3%) (57.7%) 


108 


Prediction: Earned Minimum 
Wage 


Client Earned Minimum Wage? 
No Yes 


Total 


No 


51 7 
(47.5%) (6.1%) 


59 

(54.3%) 


Yes 


9 38 
(10.1%) (35.6%) 


49 

(45.7%) 


Total i 

i 


61 46 
(57.5%) (42.5%) 


108. 



frequency 
k-total percentage 

c due to weighting, sum of numbers may not equal row/column total. 
(Continued) 
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Table 9 (continued) 



Prediction : Retention of 
Wages 


Client Retained Wages? 
No Yes 


Total 


No 


45 8 
(43.0%) (7.1%) 


53 

(50.1%) 


Yes 


7 46 
(6.5%) (43.4%) 


53 

(49.9%) 


Total 


52 53 
(49.5%) (50.5%) 


106 


Prediction: VR Satisfaction 


Client Satisfied with VR? 
No Ye_s 


Total 


No 


28 3 
(23.1%) (2.7%) 


31 

(25. S%) 


Yes 


9 81 
(7.6%) (66.6%) 


90 

(74.2%) 


Total 


37 84 
(30.7%) (69.3%) 


121 
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for basic demographics (age, sex, race) and the client's employment and 
welfare status at referral. Our findings confirm that younger (16-24) and 
older (45+) clients face greater obstacles to employment. These two groups 
were 20% less likely to achieve competitive employment and to retain earnings 
compared to those 25 to 44 years old. Their chances for achieving the 
minimum wage were even poorer, being at least 40% lower than for clients 
25 to 44 years of age. Interestingly, only younger clients showed decreased 
probability for a "successful 11 closure. 

Females and non-whites also experienced lower chances for favorable 
employment and earnings outcomes in relation to their respective counter- 
parts. In addition, whites were 22% more likely to be closed 26. The 26 
homemaker closure may account for the fact that sex bore no relation to 
closure as "successful. 11 Interestingly, males were less likely to be satis- 
fied with VR, even though they had greater chances for favorable outcomes. 
This suggests that satisfaction may be influenced by one's expectations 
the degree to which they are met or unmet. If males expect more than females 
then the chances for unmet expectations are thereby raised. It is interest- 
ing, however, that whites did not also express less satisfaction, as would be 
expected by our hypothesis. 

The above findings support the notion that younger and older people, 
females, and non-whites are harder to place in employment. Further, these 
people are more likely to lose employment over time (the last hired, first 
fired). This is indicative of current labor market conditions, which are 
largely exogenous variables over which VR has little control. However, the 
findings might justify a recommendation that VR increase efforts on behalf 
of these clients, in order that they may compete successfully in the job 
market. Still, any such decision must take into consideration the relation- 
ship which higher costs incurred through greater efforts may have on long- 
term cost-effectiveness. 

Two other groups consistently faring better on our employment and 
earnings variables were clients who were working at referral and clients 
not receiving welfare at referral. Those working at referral were anywhere 
from 18% to 65% more likely to achieve "success 11 on our measures than were 

clients not working at referral. Interestingly, clients who had worked, but 

only before becoming disabled, were least likely to achieve success. It 
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may be that the majority of these clients only recently became disabled; if 
so, they may prove more difficult to rehabilitate if they are still adjust- 
ing to their disabled condition. Corollary to this, such clients may have 
high expectations, based on what they could do before becoming disabled. 
Perhaps these clients are more prone to give up before the rehabilitation 
process can be completed. 

Clients receiving welfare at referral also were much less likely to 
achieve success in terms of employment and earnings. There are two possible 
(and not necessarily mutually exclusive) explanations for these findings. 
On the one hand, welfare recipients may have intrinsically lower employment 
potential and may represent more difficult cases. On the other hand, client.fi 
receiving welfare may have an economic disincentive to achieve success: 
these clients may seek to preserve income and especially health care and 
attendant benefits already enjoyed via public assistance, which would be 
difficult to equal given the low wage prospects often associated with com- 
petitive employment rehabilitation - 

The findings regarding clients receiving welfare and/or not working at 
referral suggest several recommendations. First, VR should perhaps expend 
greater energy on clients not working at referral. These clients probably 
need more extensive services, especially those services aimed at employ- 
ment, such as education, vocational training, and job placement. In addi- 
tion, diagnosis and evaluation should aid in effective goal identification 
and service planning. Finally, VR should identify those clients experienc- 
ing difficulty in adjusting to their disabled condition. Such clients may 
need assistance in recognizing their limitations, but, especially, VR should 
help the client become cognizant of his or her as yet untapped potential. 
In terms of clients receiving public assistance, the first need is to deter- 
mine the extent to which economic disincentives are responsible for these 
clients 1 poorer outcomes. If disincentives do explain the findings, then 
perhaps a more graduated reduction in welfare payments and health care sub- 
sidies as earnings increase might induce more favorable outcomes for these 
clients. 

Contrary to expectation, neither client education, specific disability, 
nor severity of disability showed consistent relationships to outcomes. For 
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example, we might have expected success to be linearly related to education, 
since education is generally thought to increase employability. However, 
education was significant only in terms of achieving competitive employment, 
although the relationship is consistent with theory: in relation to high 
school graduates, clients with no more than elementary school education were 
32% less likely to achieve competitive employment, while those with at least 
some college were 19% more likely to be competitively employed. Also, clients 
with an elementary education were less likely to be closed 26. 

A client's specific disability showed only sporadic relation to out- 
comes. Interestingly, clients with visual disabilities were almost twice 
as likely as other clients to earn the minimum wage. This may be because 
these clients are relatively easy to rehabilitate, particularly if they only 
require, for example, a pair of glasses to be "rehabilitated," Likewise, 
clients with drug and behavior related disabilities, who were more likely 
to achieve competitive employment, may require minimal effort to achieve 
success. The main conclusion, however, is that the specific disability of 
a client is not an important determining factor in vocational outcome. 

Finally, measures of case severity had no value as predictors of 
employment and earnings outcomes. This finding is of particular importance 
in relation to the legislative mandate for priority service to severely 
disabled clients. Although no formal definition of "priority" yet exists, 
we can be confident that the severely disabled fared no worse than non- 
severe clients, all other things being equal, in terms of one conceivable 
measure for priority (i.e., outcomes). Interestingly, both severity and 
secondary disability were related to satisfaction, although in an incon- 
sistent manner. The fact that severely disabled were more likely to express 
satisfaction points again to our "expectations" hypothesis, that, perhaps, 
they do not expect too much. However, the hypothesis is not supported by 
the responses of those with a second disability. The explanation for the 
differences in satisfaction may well be in the priority mandate for the J 
severely disabled: to the extent that these clients were receiving greater 
attention and better services, they were more satisfied. Clients with 
secondary disabilities were not a particular priority group, and it may even 
be that counselors, by focusing on the primary obstacle to employment, failed 
to meet expectations of these clients with respect to other needs. 
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The findings on education suggest that VR should form a strategy for 
dealing with poorly educated clients. One obvious suggestion might be that 
VR provide more educational services to these clients. Such a policy, 
again, must be considered in light of potential consequences for case service 
cost and length of time in service in relation to potential benefits gained. 

In comparison to client characteristics, service process variables 
showed less consistent relations to outcomes. The particular program ele- 
ments most strongly related to outcomes included: compliance with the 1973 
legislative mandate for Individual Written Rehabilitation Programs; service 
plan reliability; supervisory review of case; and certain services delivered 
to the client. Among the dependent variables, competitive employment and 
wage retention were associated with the greatest number of process variables. 
This bears out the variance partitioning findings which showed process 
uniquely accounting for a considerable amount of variance on these two 
outcome measures. However, we might have expected similar results for 
the 26 closure, for which process accounted for the greatest amount of 
variance, and achievement of the minimum wage, as well. As we will see, 
only certain key process variables were related to the 26 closure and 
minimum wage outcomes, once client characteristics were controlled. 

Possession of an IWRP impacted positively on various client outcomes. 
Clients with IWRPs were 30% more likely to be closed into competitive em- 
ployment and had 12% greater chances of retaining wages. Provision of 
IWRPs also increased, by 26%, the probability of being closed 26. Finally, 
clients with IWRPs were 15% more likely to express satisfaction with VR. 
Thus, it appears that the IWRP process (assumedly entailing client- 
counselor cooperation in service planning) has the theorized effect on 
client satisfaction. It is important to note that possession of an IWRP 
will soon be the case for all clients, assuming satisfactory agency com- 
pliance with laws and regulations, and thus will no longer be a discrimin- 
ating variable among clients. However, the finding of the positive effect 
of the IWRP is still very important now, in its relatively early stages of 
implementation, because of its confirmation that the new legislative 
requirement is sound and positive in its impact. 

Service plan reliability showed quite varying results: although 
reliability increased the chances for 26 closure and satisfaction with VR, 
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it was associated with lesser chances for achieving competitive employment 
and retaining wages. It seems reasonable that reliable service delivery 
should increase client satisfaction. Yet it is somewhat surprising to find 
reliability impacting one way (positively) on probability of 26 closure 
while impacting negatively on achievement of competitive employment and 
rett ..cion of wages. However, the explanation may well lie in the homemaker 
closure. Bivariate analysis indicates lower reliability for homemaker and 
other non- competitive closures. Apparently, these clients do not as 
reliably get their planned services, and this may be the reason for closure 
in work statuses inferior to competitive employment. 

Supervisory review of a client's case might normally be expected to 
increase chances for favorable outcomes; such review might serve to redirect 
rehabilitation strategy to more appropriately satisfy the client's rehabili- 
tation needs. However, in our sample, clients whose cases had been reviewed 
were much less likely than those never reviewed to achieve the minimum wage 
(36%) and to retain earnings (16%). In addition, the former group expressed 
less satisfaction than the latter group. Apparently, supervisory case 
review does not necessarily imply supervisor intervention for more effective 
service provision. Instead, perhaps supervisors review cases of those 
clients who are "troublesome" to their counselor: clients with unrealistic 
expectations, or who are uncooperative, or who represent particularly difficult 
cases. In fact, counselors themselves may initiate the bulk of requests 
for such reviews. 

Provision of certain VR services seems to facilitate favorable 
employment and economic outcomes. In most cases, observed effects are 
consistent with intuitive notions of service impact. Education services 
showed the greatest association with employment and earnings. In parti- 
cular, education seemed to greatly enhance a client's wage-earning power 
and ability to compete successfully over time in the job market, 
regardless of the client's personal characteristics. That is, we might 
well expect "the best and brightest clients" to fare well with education; 
we would not thereby infer that all clients should get more educational 
services since not all clients might have the aptitude to benefit from 
education. Since we have controlled for other client characteristics likely 
to affect general aptitude, the positive findings for educational services 
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do indeed indicate that most clients benefit from educational services when 
provided. It is interesting to note, also, that provision of educational 
services exerted the strongest positive effect on- minimum wage and wage 
retention; further, the magnitude of the two coefficients was very high, 
emphasizing the considerable effect of educational services. This is 
sensible, of course, since education augments a client's personal resources, 
thus providing greater opportunities. 

Other substantial services tended to impact on a client's short- and 
long-term ability to achieve and maintain employment. For example, provision 
of vocational training and job placement services increases chances for 
employment and retention. Both of these services are aimed at placing clients 
in stable employment. In addition, these two services increase chances for 
a "successful' 1 closure. Clients who received on-the-job training (OJT) or 
"other" service were more likely to retain wage. OJT may allow clients 
to gain greater familiarity with the demands of their occupations, while 
operating in a somewhat relaxed atmosphere. Those clients receiving "other' : 
services may receive auxilliary services with particular relevance to their 
occupational goals. In the case of both variables, retention should be 
enhanced. Finally, clients receiving diagnosis and evaluation, and clients 
receiving tools, have greater probabilities of achieving competitive employ- 
ment. This is sensible, inasmuch as diagnosis is aimed at identifying a 
vocational goal suited to the client's needs, and tools are provided with 
specific jobs in mind. 

Two other services -- prosthetics and family services — showed strong 
relations to some dependent variables. However, the nature of the relation- 
ships is unclear. For example, clients receiving prosthetics experienced 
far greater chances for "successful" closure, competitive employment, and 
wage retention. There is no intuitive reason why prosthetics should exert 
such an impact on outcomes, although in some cases receipt of prosthetics 
may really indicate an easy case, one requiring little effort to "rehabili- 
tate," and simple provision of glasses or hearing aid equipment. Alterna- 
tively, there may be cases where prosthetics represent a very sophisticated 
and needed service, one which is a VR specialty and, when provided, greatly 
increases employability. At this time no explanation can be given for the 
observed negative relation between family services and the 26 closure. 
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Services, in general, are very important factors contributing to 
employment outcome. Most had coefficients of very high magnitude relative 
to any of the other individual variables considered, including client char- 
acteristics. The more ''substantial 11 services, of education, vocational 
training, and job placement, showed up consistently in terms of 
quality closure and retention of benefits, although only education showed a 
strong relation to achieving minimum wages. Interestingly, the provision of 
specific services appears to have nothing to do with client satisfaction, 
except in the cases of on-the-job training and job placement services which 
showed negative relationships for which we can posit no explanation. 
Finally, it is interesting to note two things. First, provision of follow- 
up services, though infrequent, did not seem to increase the chances for 
wage retention when provided. Second, clients with many services planned 
were less likely than other clients to achieve competitive employment. Con- 
ceivably, those clients requiring many services are also the clients facing 
the greatest obstacles to employment. Alternatively, since bivariate 
analysis showed homemakers and other unpaid work statuses to have the 
greatest number of services planned, it may be that plans are changed 
and new services added when it becomes clear that the original services 
will not yield competitive outcomes. 

The remaining elements of the service process considered, time in 
process and turnover of counselors assigned to a client, showed no consis- 
tent relationships with client outcome. 

Some possible recommendations, on the basis of the above findings on 
the role of service process, include: 

(1) Agencies should emphasize completion of the IWRP process 
for each client. Although client awareness and client 
participation do not significantly affect earnings outcomes, 
they do facilitate entrance into the competitive labor 
market and retention of wages and employment and seem to 
engender client satisfaction. 

(2) Agencies may want to increase provision of certain services, 
especially those services needed for effective planning 
(i.e., diagnosis) and adjustment and entrance to the labor 
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market (such as vocational training and job placement). In 
addition, education-increases chances for success on many 
measures. 

(3"j Fuxlrher :z::vest xz£Z±nr :±s: warranted to determine: 

— d ~ caserns why c -Tp nts receiving prosthetics are so 
.vs.' ced ±x .Terms caf outcomes. It may be that these 
Hints ha. ( n^mraal need for VR services, 

— why clients' cases are reviewed by supervisors. This 
question is mostly of academic interest. However, it 
may be instructive to see more clearly whether this 
review procedure can be used by agencies to improve 
the outcomes of these apparently problematic cases. 

The last block of independent variables investigated, counselor charac- 
teristics, showed the least consistent relation to our dependent variables, 
as was earlier suggested by variance partitioning. Counselor education 
showed only sporadic association to outcomes. However, clients of two 
groups of counselors — those with no more than a college degree, and those 
with graduate degrees in psychology, sociology, and special education — 
fared worse in terms of competitive employment and earnings. In general, 
with regard to the question of how counselors with degrees in rehabilitation 
fare in comparison to others, we find that for most of our measures there was 
no significant difference. They did not appear to perform better on the 
traditional measure of 26 closure or on competitive employment or satisfaction. 
However, they did fare better than other counselors with regard to achieve- 
ment of minimum wages for clients. 

Training in rehab-related issues had varying impacts on outcomes. 
Apparently, training in caseload management bears no relation to outcomes. 
This is somewhat surprising, since one might expect such training to facili- 
tate smooth service delivery and counselor-client communication. It may be 
that the quality or content of the training available is limited. Surpris- 
ingly, training in the needs of the job market was negatively associated with 
earnings outcomes. Again, it could be that the training on this topic is out 
of date. Finally, training in the needs of the severely disabled was asso- 
ciated with greater VR satisfaction. 
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Contrary to expectations, the most experienced counselors registered the 
least success on employment and earnings measures. This may be due to greater 
enthusiasm of newer counselors — a hypothesis supported somewhat by the 
increased probability for satisfaction with VR exhibited by clients of the 
newest counselors. If this is true, then VR may face problems with counselor 
burn-out, in which case VR is well advised to address the issue for the 
benefit of all concerned. 

Counselor productivity (number of 26s) did not affect client outcomes. 
This finding should dispel, somewhat, the fears of those concerned with the 
adverse impact of pressures for high productivity, since it supports the 
contention that high productivity counselors do not necessarily produce 
poorer quality outcomes. 

In conjunction with the above finding, it is interesting to see the 
apparent positive effects of larger caseloads. It is unlikely that large 
caseloads, in and of themselves, are responsible for positive outcomes. 
Possibly, caseload size is a proxy measure for counselor skill: better 
counselors can handle more clients. Likewise, better counselors have 
greater success. 

Finally, clients of relatively autonomous counselors (i.e., those 
experiencing less direct supervision) were 20% less likely to earn the 
minimum wage. Autonomy did not, however, have a significant relation to 
other outcomes, including probability of 26 closure. Thus, while clients 
of such counselors are equally likely to find and retain employment, their 
counselors may be freer to close them into less well-paying jobs. To the 
extent that this is a concern, agencies may want to employ more supervision 
of counselors. 

To summarize, certain conclusions and recommendations are suggested by 
the analysis of counselor characteristics. First, VR need not be too con- 
cernedwith a counselor's specific education. Second, more detailed analysis 
of the nature and extent of in -house training is required. Current training 
programs, with the exception of training about the severely disabled, do not 
appear to be particularly fruitful, but it remains possible that improved 
curricula could change this situation. Third, VR may face the problem of 
counselor burn-out, a problem with serious consequences for program perform- 
ance. Fourth, high productivity does not seem to be incompatible with 
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"quality 11 rehabilitation. Finally, agencies may wish to provide greater 
counselor supervision in order to facilitate achievement of employment 
and earnings "suitable to the clients 1 capabilities. 11 



54 



05 



ERIC 



51 



III. SITE- OR AGENCY- LEVEL MACRO ANALYSIS 



Agency- level analyses are at the front line of evaluation research 
for data-based decision support. Agency-level analyses address the two 
most important questions for evaluation research on the standards: 

• how is attainment on the various standards data elements 
related across the state VR agencies? and 

• what explains differences among state VR agencies in attain- 
ment on specific standards data elements? 

Both of these questions are discussed further below. 

RELATIONS AMONG THE STANDARDS DATA ELEMENTS 

Description of Approach 

The relations among the program standards are a major concern for the 
standards system. The statistical methodology for this type of analysis 
is most appropriately that of correlation and factor analysis. First, the 
analysis of the data elements is done standard by standard. For this 
analysis, correlations are sufficient. Then, factor analysis is used to 
explore the dimensionality of all the data elements of all the standards 
at once (although a separate analysis might be appropriate for the Perfor- 
mance Standards and for the Procedural Standards). 

The factor analysis technique BPA recommends to explore the dimension- 
ality of the measures is principal factoring with iteration for improving 
the estimates of the communalities . The choice of the number of factors 
to retain is to be made using Guttman f s strongest lower bond. Using this 
criterion, the initial number of factors is chosen based on the number of 
components with eigenvalues greater than or equal to ■ in a principal 
components analysis. Then, an initial principal factoxxng with iteration 
o£ this initial number of factors gives the final number of factors, those 
with eigenvalues greater than or equal to one. A second principal factor- 
ing with iteration gives the initial factor matrix of loadings. To aid the 
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interpretation of these factor loadings, an orthogonal — rather than 
oblique — rotation, varimax rotation with Kaiser normalization, is per- 
formed. Oblique rotation is not used because of the difficulty of choos- 
ing the degree of correlation allowable between the factors and thus the 
question of whether highly correlated factors are really separate factors 
at all. The missing data technique proposed is pairwise deletion. (See 
Harman for an elaboration of the factor analysis technique and of the 
methodological decisions made.) 

Case Example: Relations Among the Program Standards 

The number of performance standards data elements (21) creates cer- 
tain problems in designing and implementing the standards system. The 
problems revolve around which standards and elements to emphasize, real- 
izing that some choice will have to made between improving on one stan- 
dard over another, or improving a little on each. The problem is exacerbated 
when improvement on one standard may be at the expense of another standard, 
i.e., with decline on another standard. 

The various data elements represent diverse goals, sG.7\e in conflict. 
With a single objective function (single goal), VR may be expected to 
maximize on the measure (e.g., close as many clients as "26s" as possible). 
But with multiple objectives, as represented by the standards and data 
elements, conflicting in many subtle and not-so-subtle ways, a system of 
attainment levels can only indicate desired achievement on all elements. 
Among elements are trade-offs, however. For example, an agency might 
maximize its benefit-cost ratio by reducing its coverage rate and by cream- 
ing. As a result, success in achieving one program goal could be counter- 
productive to success on other goals. 

The reason this problem arises, of course, is because the basic VR 
program has several conflicting, although legitimate , performance and 
service objectives. (To paraphrase one regional official visited during 
our study, VR is really many programs, each with distinct goals.) 

The discussion below presents some empirical findings on the relations 
among measures of performance to underline the difficulties that multiple 
measures cause. Then, two approaches to the problems of multiple measures 
are presented with their shortcomings noted. Last," the implications of 
continuing with multiple measures for the standards system are laid out. 
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A data base of all the standards data elements is not available for 
all agencies (since several new data collection instruments are involved 
and since the standards 1 pretest was carried out in only six state VR 
agencies). However, the data are available for some of the data elements, 
allowing analysis of some of the relationships among VR performance 
measures. 

In Knuce, Miller, and Cope, relationships among several measures of 
inputs, process, and outputs were investigated for the 54 states and U.S. 
territories. Several of the measures used are found in the standards data 
elements, and others are very similar. The bivariate correlations among 
some of these measures for 1968 and 1969 are shown in Table 9. The authors 
state : 

Both the high levels of rehabilitation rate and rehabil- 
itant^ salary are desirable program outcomes. Yet, the 
results suggest that these outcomes may. be incompatible with 
each other. Such an incompatibility is highlighted by the 
opposite relationship that the two output variables (the 
rehabilitation rate and rehabilitant 1 s salary) have to the 
number of cases served. Where more clients are served, the 
rehabilitation rate is higher but rehabilitant 1 s salary is 
lower. Conversely, when fewer clients are served a higher 
placement level (rehabilitant ! s salary) is achieved. Paren- 
thetically, it is noted that volume, as measured by number 
of cases per 100,000, is associated with lower rehabilitation 
cost. 

The inverse relationship between rehabilitation rate and 
rehabilitant ! s salary has special implications for program 
evaluation. Many programs that look good on one of these 
outcome variables will look bad on the other one. This 
finding does not necessarily imply that programs high or 
low on either of these variables are good or bad. However, 
the results do strongly support a position that the two 
kinds of programs have different resources and strategies. 
Those agencies with high rates tend to have more financial 
resources, work with more clients, rely more on workshops, 
and keep clients in the caseload for a. shorter period of 
time. Those with lower rates tend to deal with fewer clients, 
be more selective in accepting clients, keep them in the 
program for a longer period of time, and provide them more 
training. Therefore, examination and evaluation of a program 
on the basis of only one criterion could lead to erroneous 
conclusions about program effectiveness, [page 137] 

In Dodson (1978), factor analysis was used to investigate the relation- 
ships among measures of outputs, and among measures of performance for the 
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Table 9 

Correlation Among Measures — Knuce, Miller, and Cope 



Measure 


1 


2 


3 


4 


1. Clients served per 
100,000 population 




-.38/-.43 a 


.89/. 92 


-.34/-. 29 


2. Expenditure per 26 
closure 






-.39/-. 56 


.06/. 30 


3. Rehabilitation rate 
per 100,000 popula- 
tion 








-.39/-. 39 


4. Average earnings at 
closure for 26 

1 











a l968/1969 
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states for 1970. The results of the factor analysis for the performance 
measures are shown in Table 10. 

A factor analysis of all eight performance measures yields three 
factors. The first has high loadings on percent with earnings at closure 
(.93), competitive employment (.86), and homemaking (-.93). The second 
factor has high loadings on earnings at closure (.93) and on increase in 
earnings (.81), reflecting the high correlation (.92) between these two 
measures. The third has a high loading on benefit cost ratio (.89). 
These factors account for the following percentages of the shared varia- 
tion: 50.6%, 31.1%, and 18.3%, respectively. Overall, these three factors 
account for 71.1% of the total variation. Of the individual measures, 
only reduction in public assistance (4.5%) and percent with public assis- 
tance at closure (23.1%) have less than 50% of their variation explained 
by these three factors. 

Thus, the percentage of earnings or competitive employment (as in 
data element 4i or 5i) comprises a very different dimension than the 
absolute level of earnings (as in data element 4ii). The cost benefit 
ratio presents again another dimension.* 

The main conclusions to be drawn from these analyses are: 

• some of the standards data elements are positively related, 
so that an agency doing well in one data element will likely 
be doing well on other data elements; 

• some of the standards data elements are unrelated, even 
within the same standard, so that an agency doing well on 
one data element will not be related to its doing well on 
other data elements; • 

• some of the standards data elements are negatively related, 
so that an agency doing well on one data element will likely 
do less well on another data element; and 

• the attainment of VR agencies with regard to the Performance 
and Procedural Program Standards is clearly ^([^dimensional. 



'These dimensions have a correlation of zero, since orthogonal 
rotation has been performed; when oblique rotation was tried, the cor- 
relations between dimensions stayed near zero. 
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Table 10 

Factor Analysis of Output Measures -- Dodson (1978) 





Factor 


Measure 


1 


2 


3 


1. Percent 26 with earnings at closure 


.93 






2. Average earnings at closure 




.93 




3. Percent 26 with competitive employment 


.86 






4. Percent homemaking 


-.93 






5. Increase in earnings from referral 
to closure 




.81 




6. Reduction in public assistance 








7. Percent with public assistance at 
closure 








8. Benefit cost ratio (crude) 






.89 
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Some possible approaches to the problems of the relations among program 
goals are described in Berkeley Planning Associates, Vocational Rehabil- 
itation Program Standards Evaluation System: Final Report, Volume II: 
Using the System: An Analytic Paradigm for Management , pp. 54-60. 

EXPLAINING DIFFERENCES IN ATTAINMENT AMONG AGENCIES 

Description of Approach 

Evaluation research may be needed to explain differences in attain^, 
ment among the state VR agencies in any of the following scenarios in the 
operation of the data-based decision support system: 

o federal (or state) program managers, in investigating problem- 
atic attainment for a particular agency (or unit) , are unable 
to uncover a set of possible problems; 
© program managers can identify alternative problems, but they 
are unable to choose between them; 

• program managers can identify possible problems, but have 
difficulty suggesting corrective actions; or 

• program managers notice .come patterns or trends over time in 
problematic attainment across state VR agencies (or across 
sub-state units). 

For example, if the explanation for problematic attainment for cost 
per 26 closure for a particular agency (or sub-state .unit) was that no m c 
enough clients were being closed successfully (but enough clients were 
being- closed) , an analysis examining what led to the lower success rate 
would be needed. A first step would be an agency analysis, conducted for 
the eighty-odd agencies (or some subset such as blind agencies). 

The structure of this agency-level or site-level analysis is also very 
simple, like the client-level analysis. The analysis attempts to explain 
a dependent variable, such as percent of.clients closed successfully, by 
a set of independent variables. These independent variables should include 
controlling variables, such as environmental conditions, but the indepen- 
dent variables most importantly must also include intervention variables, 
variables that RSA and the state VR agency can manipulate. By including 
the intervention variables, the explanations suggested by the analysis can 
easily be turned into corrective actions. 
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Existing Agency-Level Analyses 



There have been several agency-level analyses conducted on the VR 
system. However, most of them have been concerned with very specific 



questions (e .g. /"Collignon and Serot attempted to answer the single ques- 
tion of whether the organizational structure of the state VR agency has 
any impact on performance), have used only a few measures of the VR sys- 
tem (e.g., Flynn uses only six measures) , or have used primitive statis- 
tical methodology (e.g., Knuce, Miller, and Cope use bivariate correlations 
only). However, there have been some more rigorous efforts undertaken 
(e.g., Rubin). The BPA approach to doing agency-level analysis is based 
on a series of research pieces by Dodson. In Dodson (1973), the model- 
ing approach and statistical methodology are fully developed, and the 
application of the model and of the statistical methodology are applied 
to a limited (in the measurement sense) example. In Dodson (1978), the 
measurement of the VR system and the theoretical underpinnings are more 
fully worked out. In Dodson (1979), the methodology is applied to the 
operations of the SSDI/SSI-VR Special Program. 

A Proposed Macro Model 

The proposed "macro" model for conducting agency-level analyses to 
explain differences in attainment across state VR agencies is given in 
Figure 2. Note that the macro model builds on the micro model. The 
model components are shown to be: environment , government , service agency , 
clients , inputs , outputs , and performance . A short definition of each 
model component follows. The environment is the economic, social, and 
physical environment. The government is the political system, with its 
structure, process, and expenditures and policies. The service agency is 
the set of characteristics of administrative structure, program structure, 
method of service delivery, personnel and expenditures, non-direct services, 
client selection, effort, and growth. The clients component includes demo- 
graphic and economic characteristics of the clients, as well as the char- 
acteristics of their problems requiring services. The inputs component 
includes the amounts, types, mix, and interactions of direct services, as 
well as the quality and appropriateness of serivces. The outputs component 
incJuc^s throughput, processing efficiency, output production, and administrative 
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Figure 2 
Proposed Macro Mode 1 
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success. The performance component refers to program goal achievement. 
Both the outputs component and the performance component refer to the 
success of the service agency, but success along different dimensions. 

The causal hierarchy (what causes what) is as is seen in the model: 
environment, government, service agency, clients and inputs, outputs, and 
performance. The environment affects all parts of the system. Different 
types of governments and service agencies occur in different environments, 
partly in response to these environments. Different clients come from 
different environments and the types and amounts of inputs are constrained 
by the environment. And the success of the system — its output and per- 
formance is affected by the environment. In turn, the type of govern- 
ment affects the structure of the service agency, the types of clients 
served, and the kinds of inputs that are provided, as well as the success 
of the system — its output and performance. The kinds of clients served 
are influenced by the structure of the service agency and by the types and 
amounts of inputs offered, as well as by the environment and government, 
as noted before. In addition, the inputs are determined by the kinds of 
clients to be helped and the service agency, as well as by the environment 
and government, as noted before. The outputs are determined by the types 
of clients served, the inputs, and the service agency, along with the 
environment and government, as noted before. The performance of the 
system is affected by this same set of factors, with the addition of out- 
puts. Each of the components is discussed further below. 

To the environment component in the micro model is added the concept 
of population-at-risk. Population-at-risk here refers to those who could 
benefit from the services of interest. Similar concepts are need, latent 
demand, and problem incidence and prevalence (a more suitable concept than 
incidence in a cross-section analysis). Not only does the population-at- 
risk concept suggest the dimensions of the possible service population, 
but also illuminates a very important part of agency and counselor behavior 
client selection. Clients may be selected on the basis of available re- 
sources, of first-come/first-served, of balanced caseload, of greatest need 
for services, of least cost, of fastest throughput, or of most likely to • 
benefit from. vocational reahbilitation. Creaming refers to the skimming 
of easier cases (least cost, fastest throughput, most likely to succeed 
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administratively). Note that the concept of effective demand is not im- 
plied here, since these services arenot paid for. Also, the concept of 
demand itself is not implied here, since a measure of demand for services 
is rarely available. Usually, needs assessment or prevalence estimation 
is based on demographic projections and is used to identify the number of 
people who could benefit from the social service delivery system or who 
have the problems of interest. 

To the micro model components must be added the macro model com- 
ponent of government , both general and specific government. In a macro 
model not only must the services given to individual clients be understood, 
but also the resources given to the service agency must be understood. 
Moreover, the service agency itself, and the inputs, the clients, the out- 
puts, and the performance are all affected by government. A very large 
number of possible measures of this general concept of government exist. 
However, in Dodson (1978) this very large number of measures of the govern- 
ment were analyzed, and the following six measures stood out as influencing 
performance: number of legislative bills, electoral structure, expenditures 
per capita, bureaucratic merit, percent state revenue from the federal govern- 
ment, and the congruence of state policies with public opinion in the state. 

The s ervice agency component is the same as for the micro model. 
However, the emphasis in a macro model is on the behaviors of service 
agencies, vis-a-vis client selection, choice of services, and the emphasis 
on output versus performance. 

One part of the clients component of the micro model, case difficulty, 
requires an aggregated procedure for the macro model, if a single index 
is desired. Weighted case closure is a method that, attempts to take into 
account client difficulty when comparing output or performance measures 
(see Conley, 1973; Sermon), although the method is somewhat more oriented 
to the level of individual counselor performance than to agency-level per- 
formance. 

In the macro model, a more general component of inputs is used, 
rather than services, since the actual services are very difficult to 
measure in VR. In addition to the other aspects of inputs, productivity 
of inputs is important in a resource allocation-oriented macro model. 
Unfortunately, little is known about the productivity of inputs, except 
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in the crude terms of resulting processing efficiency. Also, when dis- 
cussing macro modeling, the reliability of services is important, 
although conceptually reliability is included as part of outputs in the 
macro model. Reliability asks whether the services outlined in the 
plan are actually given, and given in the correct sequence, mix, and 
amount. Although reliability may, be defined for the individual client, 
at the micro level the measurement is either yes or no; only for the 
macro level can reliability be used in the probability sense. Unfor- 
tunately, information about the plan realization is not available on the 
R-300. Another important dimension of macro inputs is the integration 
of services; however, no information is available on the R-300 for this 
dimension either. 

The process results component of the micro model is broken into 
two parts for the macro model, outputs and performance, to distinguish 
processing efficiency from goal achievement. Outputs , or processing 
efficiency, includes from the micro process results: acceptance or 
rejection for services, completion or non-completion of process, reliabil- 
ity of the plan for services, administrative success, client involvement 
in planning and in service delivery, and time in process. In addition, 
the concept of output production is used to capture the amount of output 
per unit of input, input defined in terms of counselors or dollars. 

Performance , or goal achievement, includes from the micro process 
results: return for services, level of outcome, change in level of out- 
come, quality of outcome, persistence or permanence of outcome, level of 
public support still required, release from or continuance in an institu- 
tion, client satisfaction with services, personal development, and increase 
in group awareness and group organizing. In addition, the concept of cost 
effectiveness is used to capture the goal achievement, in terms of change 
in earnings, per unit of input, in terms of cost. The meaning of performance 
here is rather different (see Bennett and Weisinger, pp. 61-88; Harris; 
Walls and Tseng for other measures of performance in VR) and should be 
kept in mind. Also, the meaning of performance here is not the same as 
in Performance Standards. 
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Case Example: An Explanation of Differences in Attainment in the SSDI/ 
SSI-VR Special Program 

The previous discussion of the proposed macro model has been largely 
theoretical. But, once again, what will such an analysis look like in the 
data-based decision support system? Below is an example of explaining 
differences in attainment among state VR agencies in their operation of 
the SSDI/SSI-VR Special Program. 1 Several of the measures used are either 
incorporated into the standards data elements, or similar to some of the 
standards data elements. This example illustrates the hypotheses examined, 
the measures used, the analytical and statistical tehcniques employed, and 
the types of conclusions that can be drawn from such an analysis. First, 
a set of conceptual factors affecting performance, along with measures 
for each of these factors, are discussed. Then, empirical relations 
between these factors and four types of performance are examined. 

Measures of Performance 

The four performance measures selected for analysis reflect an interest 
in cost-benefit performance as well as administrative performance. The 
four measures include: 

• Coverage, or the percentage of SSDI and SSI recipients (the 
population-at-risk) served; 

• Rehabilitation rate , or the percentage of all closures who are 
closed in status 26; 

• Percent SGA , or the percentage of 26 closures with weekly earn- 
ings at or above the substantial gainful activity (SGA) level; 

and f 

> 

• Cost per SGA , or the total program expenditures divided by the 
number of SGA closures. 

Factors Affecting Performance 

Economic Environment . The economic environment within the state must 
surely determine how easy it is for the individual client to find 



Drawn from Dodson, 1979. 
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employment. In this analysis, measures of unemployment and of wages are 
used to capture the economic environment. State agencies facing an economic 
environment of high unemployment and/ or low wages are expected to perform 
less well on the measures of economic performance, such as percent with 
earnings at substantial gainful activity level (SGA) . 

However, there are three problems with these two measures of the 
economic environment. First, these measures are for the entire state, not 
for the local labor market. The economic environment does vary quite a 
bit within states, and measures of unemployment and weekly wages are not 
available for local labor markets, only for certain cities. Moreover, 
matching clients with specific labor markets is very difficult. 

The second problem is that these two measures are for the general 
population. The job market for the disabled may be similar to the job 
market for the nondisabled, but probably is not. The job market for the 
disabled may include underemployment, secondary labor markets, sheltered 
employment, low wages, concentration in specific sectors and industries, 
and discrimination. Measures of the job market faced by the disabled 
are not available and, thus, the measures of economic environment may not 
represent the economic environment of the VR agency. 

Third, the only available weekly wages data are for manufacturing 
industries. Several have criticized the use of such in examining VR agencies 
both because the service sector is not represented and because there are 
certain states where manufacturing is such a small part of the industrial 
mix. 

. Benefit Levels . States vary widely in the average monthly benefits 
that they provide, both due to differing beneficiary populations and due 
to differences in state supplementation (for SSI beneficiaries). The 
higher the benefit level, relative to wages and cost of living, the more 
likely clients will choose continued beneficiary status over employment. 
Average benefit levels can function, then, as a crude measure of disincen- 
tives. 



VR Agency . The characteristics of the VR agency delivering services 
should affect the performance of these agencies. The characteristics 
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considered here are the organizational overstructure of the VR agency, 
whether the agency is a blind or general agency, and whether the agency 
is large or small. 

The definitions of the alternative organizational overstructures are 
derived from the definitions used by RSA. There are three basic forms of . 
organizational overstructure for the agency administering rehabilitation 
programs. These are; 

(1) independent agency reporting directly to Governor or Legis- 
lature; 

(2) bureau within a broader functional agency. In rehabilitation, 
when bureaus are the organizational unit responsible for 
administering programs, the bureaus — due to the historical 
origin of the program have traditionally been lodged under 

a State Education or Vocational Education Agency or Department; 

(3) agency under a multifunctional "umbrella 11 state agency, which 
can include welfare, labor, social service, and/or health 
functions . 

Three basic conceptual arguments have been raised in favor of includin 
rehabilitation agencies under umbrella agencies in recent years. First, 
proponents of umbrella agencies argue that this form of overstructure 
produces greater coordination of related services (such as vocational 
rehabilitation, education, welfare, medical services, and employment ser- 
vices) and thus increases the effectiveness of functional service programs 
in achieving their objectives. If each service is provided by a separate 
agency, there exists the potential problem of fragmentation of services, 
duplication of services, and ineffectiveness as clients can get lost in 
trying to find the appropriate services in a confusing array of agencies. 
Merging agencies under a single structure presumably ensures improved 
coordination of services such that clients and their families receive the 
full mix of services necessary to achieve society 1 s objectives, and thus 
the effectiveness of each particular service is increased. Second, pro- 
ponents of umbrella agencies maintain that this form of overstructure can 
increase the efficiency of service delivery by permitting a significant 
saving of overhead expenditures, especially administrative expenditures. 
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Third, proponents argue that inclusion of all social service or manpower 
programs under umbrella agencies facilitates efficient resource allocation 
across programs, quite apart from better management and resource allocation 
within programs. Thus, the overall funding level of rehabilitation programs 
as compared to other programs might change, depending on the relative 
performance of rehabilitation and other programs in achieving goals desired 
by the state. 

On the other hand, supporters of other forms of organizational over- 
structure have maintained that an umbrella agency could become too large 
and unwieldy to properly provide services to a client. The individual 
client may become lost in a large mass of clients having diverse problems 
and service needs. Increased red tape resulting from efforts to coordinate 
services may impede flexibility and delay services, encumber professionals 
in paper work rather than service delivery, and reduce the effectiveness 
and efficiency of service delivery. Moreover, some argue that the creation 
of umbrella agencies adds unnecessary layers of bureaucracy which increase 
rather than reduce administrative costs and which disrupt without compen- 
sating benefits the management of the specialized systems which have 
gradually evolved over many years to respond to the particular needs of 
client subpopulations . Moreover, VR agencies lodged under an umbrella 
agency may lose sight of the traditional goals and focuses of the VR agency, 
especially with respecr to restorative services and vocational goals. 

Blind agencies and general agencies (including combined agencies) are 
quite different. The blind agency serves only one type of client and is 
very specialized. These agencies are usually very small and the service 
milieu is more intensive. Moreover, blind clients are very different from 
other types of clients, in basic characteristics as well as in employment 
goals. 

• Large versus small agencies is a question of both the quality of 
service as well as of economies of scale. Many would argue that the small 
agencies are able to offer more personal and more intensive service 
delivery. On the other hand, many argue that the larger agencies are able 
to offer economies of scale. 
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Expenditures , The way VR agencies choose to spend their special 
program funds also will affect how well they will perform. 

The first decision made, either explicitly or not, is whether to 
spend the allocation on many clients or on a few clients. An easy way to 
measure the way this decision has gone is to look at the total expenditures 
divided by the number of clients served, to get an average expenditure per 
client served. 

The second decision made, either explicitly or not, is how to divide 
the monies among the major expenditure categories: administration, coun- 
seling and placement, and services to individuals. Because the percen- 
tages of these categories must add to 100% for any particular agency, one 
of the categories must be left out of the analysis, as is services to 
individuals for this analysis. 

Staffing. A major area of concern in the special program is the use 
of specialists, including supervisors and especially counselors. Argu- 
ments on both sides abound, and VR agencies change from specialist to 
non-specialist and back again over time. 

No survey of state agencies in this area was available, so a proxy 
is used whether or not any staff was paid for out of special program 
funds . 

Referral . A particular complaint of VR agencies has been the paper- 
work required for the processing of the large number of referrals from 
the State Disability Determination Unit and the Social Security District 
Offices (SSDOs). An equally important question, however, is whether 
these referrals lead to successful closures (see, for example, Mueller 
and Patrick). For this reason, the percent of referrals from DDS or SSDOs 
has been included as a factor affecting performance. 

Use of the Special Program . Both SSDI and SSI beneficiaries can be 
served either in the special program or in the general 110 program. 
Agencies may elect to cream the best beneficiaries for the special program, 
i.e., to choose the clients most likely to achieve SGA closure or termina- 
tion. Other agencies may elect to serve more beneficiaries in the special 
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program, thereby getting additional monies. The percent beneficiaries 
served in the special program is thus included to reflect this choice 
by a VR agency. 

Demographic Characteristics of Clients . In all analyses of the per- 
formance of agencies delivering services, the characteristics of clients 
must be controlled for, so that differences in performance will not be 
attributed to services, for example, when the differences are really due 
to the type of clients served. The basic categories of age, education, 
sex, and race are included here. Age and education have been combined 
into one variable because of the correlation between the two. The 
categories, between 25 and 44 years old and with 12 or more years of 
education, are drawn from the Rutgers analysis (Berkowitz, et al .), of 
which clients are most likely to terminate. 

Economic Situation of Clients . For the economic situation of clients, 
the percent with dependents, the percent with earnings at referral, and 
the percent with public assistance at referral have been chosen to repre- 
sent the various factors affecting employment. 

Disabilities of Clients . A disability by disability analysis would 
require a micro analysis, because of the many different disability groups. 
Here, an RSA measure of insubstantial handicaps to employment is used 
(RSA, 1975). The attempt is to get a measure of client difficulty, using 
disability codes. Thus, the percent with insubstantial employment handicaps 
reflects the percent of less difficult clients, since few are in the specified 
disability codes. These codes are the absolute easiest cases, and do not 
allow differentiation among the moderately difficult and very difficult 
clients in the special program. Moreover, the R-300 severely disabled 
category is of no help here, since SSDI and SSI clients are by definition 
severely disabled under the R-300 classification. Also, because the blind 
codes are not included in this definition, blind agencies have zero percent 
with insubstantial employment handicaps. Thus, this measure and the blind 
agency measure are necessarily colinear. 
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Services . A service by service analysis would also require a micro 
analysis, because of the many different services and different service 
combinations. Here the major service categories of diagnosis and evalua- 
tion, restoration, and training are used. The categories omitted are 
maintenance, services to family members, other services, and post -employment 



Time in Process . Very few process measures are available from the 
existing data sources. For example, IWRP, counselor contact, reliability 
of service delivery are not available. The only readily available process 
measure is time in process. Here, two measures are used, months in refer- 
ral and months from referral to closure. Arguments for the efficacy of 
both faster through-put and more lengthy service delivery exist. 



The measures of the factors affecting performance are here used to 
examine the empirical relations between these factors and the four measures 
of performance . A summary of the measures of the factors affecting per- 
formance is given in Table 11, along with definitions of the measures. 

The statistical methodology used to examine the empirical relations 
is that of OLSQ regression. 

For each of - \o four measures of performance, regressions were run 
for FY 75 and for FY 76, separately for SSDI and for SSI clients. In 
addition, regressions were run for general agencies and for all agencies. 
No regressions could be run for blind agencies alone, since there were 
too few blind agencies (given the number of independent variables). 

The results from these regressions are given in Tables 12 through 19. 



services. 
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Table 11 

Definition of Measures of Factors Affecting Performance 



Measure 


Definition 


Source 


Economic Environment 






(1) unemployment rate 

(2) average wages 


\ unemployed of total work force in state 

average weekly earnings of production workers 
in manufacturing industries in state 


Statistical 
ads tract 

Abstract 


Benefit Levels 






(3) average benefit 


average benefits for state; for SSI, federal 
plus state supplementation; for blind 
agencies, blind SSI; for general agencies, 
disabled SSI; for combined agencies, average 
of blind and disabled SSI (weighted by 
number of beneficiaries) 


Statistical 
Supplement 


VR Accncv 






(4) education agency, 
umbrella agency, 
independent agency 

(5) blind agency, 
general agency 

(6) large agency, 
small agency 


administrative location and overstructure 
of VR agency 

whether blind or general agency 

based on number of clients served in 110 
program; large general or combined if 
25,000 or more; small blind if 1,000 or 
more; based on distribution of agency 
sizes to give .5,. 5 distribution 


IM-72-60 

Program Data 
Progrcvm Data 


Expenditures 






(7) expenditure per 
client served 

(S) * expenditures on 
administration 

(9) V expenditures on 
counseling and 
placement 


total expenditures by state VR agency from 
DI Trust Fund and from General Treasury 
under SSI special program -f- number clients 
served (i.e., clients in active statuses 
10-24 and clients closed) 

% expenditures in (7) spent on administration 

\ expenditures in (7) spent on counseling and 
placement 


Program Data 

Program Data 
Program Data 


Staffing 

(10) specialist staff, 
no specialist 
staff 


whether any special program monies expended 
on staff 


Program Data 


Referral 

(11) \ DDS-SSDO 
referrals 


% 26 closures with referral at any time from 
DDS or SSDO 


R-300 tapes 


Use of Special Program 

(12) *. beneficiaries 
served in special 
program 


\ SSDI or SSI beneficiaries served under 
special program of all SSDI or SSI 
beneficiaries 


IM-76-32 



/•o 
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Table 11 (cont.) 



Measure 


Definition 


Source 


Clients Demographics 








(13) 


*• between 25 and 44 


a 


rlncnrnc hnfuppn inrf d*I VMTS old and With 12 
LIUjUI L3 ULknCLlI feaJ tlllU *»M j till j wau uiiu nitit A ^ 


11-300 tapes 




years old and with 




or more years of education 






12 or more years 










of education 








(14) 


% male 


% 


26 closures male 


R-300 tapes 


(15) 


# white 


"0 


26 closures white 


R-300 tapes 


Clients Economic 








Situation 








(16) 


*• with dependents 


% 


26 closures with one or more dependents 


R-300 tapes 


(17) 


• with earnings at 


% 


"}f\ rlncnrpc with avprsfp wppklv paminps 

£U L IUjUICj WILD BVCia^C Jr cai 111115 J 


R-300 tapes 




referral 




aT r^fprra 1 orpaf pr f linn f\ 




(18) 


* with public 


* 


26 closures public assistance recipients 


R-300 tapes 




assistance at 




at referral 






referral 








Clients Disability 








(19) 


*• with insubstan- 


0 

% 


26 closures with insubstantial employment 


R-300 tapes 


tial handicaps 




handicaps , defined as % 26 closures with 








disability codes of digestive system disorders, 










hearing other than deafness, character, 










personality, and behavior disorders, rnild^ 










mental retardation, and hay fever/asthma 




Services 








(20) 


*i services spent 


% 


special program expenditures for services 


Program Data 


on diagnosis and 




to individuals spent on diagnosis and 






evaluation 




evaluation 




(21) 


* services spent 


% 


special program expenditures for services 


Program Data 


on restoration 




to individuals spent on physical and mental 










restoration 




(22) 


\ services spent 


a. 


special program expenditures for services to 


Program Data 




on training 




individuals spent on training 




Tine 


in Process 








(23) 


months in referral 


average months in referred statuses (00,02) 


R-300 tapes 






for 26 closures 




(24) 


months from 


average months from referral to closure for 


R-300 tapes 


referral to 




26 closures 






closure 









Sources : 

1. IM-72-60 -- RSA. "State Agencies Responsible for Administering Vocational Rehabilitation 
Programs. 11 Information Memorandum RSA- IM-72-60, March 10, 1972. 

2. IM-76-32 -- RSA. "Quarterly Status Report: Disability Insurance Program (SSD1) and Sup- 
plemental Security Income Program (SSI) , Fiscal Year 1975." Information Memorandum RSA- 
1M-76-32, October 30, 1975. Also for FY 76, RSA- IM-77-6, October 29, 1976. 

3. Program Data -- RSA. "State Vocational Rehabilitation Agency: Program Data Fiscal Year 
1975. ".Also, Fiscal Year 1976. 

4. R-300 tapes computer tapes of all closures during FY 1975 and of all closures during 
FY 1976 with Federal Special Program Identification U-300 codes of SSDI or SSI special 
program, computer tapes provided by RSA. 

5. Statistical Abstract — Bureau of the Census. Statistical Abstract of the United States . 
For 1976 and 1977. 

6. Statistical Supplement — SSA. "Annual Statistical Supplement" to the Social Security 
Bulletin. For 1974 and 1975. 



76 



72 



Table 12 

Regressions; SSDI Clients; Dependent Variable: Coverage 



! 


lib t J 1 




p y i ,\ - - r?j\c ics 


Independent v, ..U'lc 


cocf f ic icnt 


t-valuc 


coefficient t -value 


FY75/FY76 


FY75/FY76 


FY75/FY76 FY75/FY76 


Econonic Environnent 








unemployment rate 
average wages 

Benefit Levels 
average benefit 


.04/. 29 
.004/. 005 

-.009/-. 030 


.16/. 83 
.21/. 24 

-.18/-. 44 


[Coverage for SSDI clients 
is not defined for blind 
agencies and thus no regres- 
sion on all agencies is 
possible . ] 


VR Agency 








education agency 
umbrella agency 
independent agency t 


.04/- 1.63* 
-1.27/-1.46 


.04/-1.32 
-1.18/-1.18 




blind agency 
general agency t 


X 


X 




large agency 
small agency t 


-1.06/-1.41* 


-1.15/-1.35 




Expenditures 








expenditure per client served 
% expenditures on administra- 
tion 

% expenditures on counseling 
and placement 


X 

*.16/-.10 
-.02/-. 06* 


X 

1 .52/-. 80 
-.47/-1.37 




Staffing 








specialist staff 

no specialist stafft 


-.34/-. 25 


-.30/-. 22 




Re fcrral 








\ DDS or S5DO referrals 


.02/. 01 ' 


1.02/. 29 




V'f* S^r : ,il Program 








•4 b; Served in 

special program 


♦.07/.04 


2. 15/. 35 




intercept 


-.57/9.32 






R 2 


.195/. 196 






R* 


-.032/-. 036 






F 


.86/. 84 






significance of F 


.584/. 599 






n 


51/50 






min n 


51/50 







No: 

* significant at .20 level or less two tailed test of significance for null hypothesis that coefficient 
is zero 

t base category for a dummy variable -- category to which coefficients are compared 

X variable excluded from the regression: blind agency is excluded when only general agencies are being 
analyzed; *• with insubstantial handicaps is excluded when all agencies arc being analyzed because of 
collincarity with blind agency — sec text; expenditure per client served is excluded from coverage and 
cost per SGA dependent variable regressions because of shared terms in the definitions -- see text. 
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Table 13 

Regressions; SSI Clients; dependent Va : - "'Me: O v erage 





General Agencies 


All Agencies 




Independent Variable 


coefficient 


t-valuc 


coefficient 




t-value 




FY75 '76 


FY75/FY76 




FY75/FY76 


Econonic Environncnt 












unemployment rate 
average wages 


.17/.42* 
.003/. 006 


.86/1.41 
.23/. 46 


-.30/. 38 
.042/. 041* 




-.60/. 64 
1.29/1.59 


Benefit Levels 












average benefit 


-.018/-. 051 


-.96/-1.92 


.034/-. 050 




.68/- 1.04 


VR Agency 












education agency 
umbrella agency 
independent agency t 


-.54/-1.89 
♦-1.34/-1.47 


-.58/-1.72 
* -1.49/-1.52 


1.59/. 44 
2.02/2.25 




.61/. 18 
.92/1.08 


HI 4 nr! flffPnrv 

Ul 11IU IlKCII^ jr 

general agency t 


v 

A 


Y 
A 


1 o . 71/10. 52 




H • OO/ *♦ • UO • 


large agency 
small agency t 


*-1.29/-1.10 


-1.64/-1.19 


-.76/. 83 




-.39/. 46 


Expenditures 












expenditures per client served 
% expenditures on administra- 
tion 

% expenditures on counseling 
and placement 


X 

* i a / no 
.02/-. 02 


X 

i ci / an 
.72/-. 67 


X 

* - . 30/- . 02 
.08/-. 02 




X 

-1 /- 11 
- 1 . yo/ -.ii 

1.15/-. 31 


O I ll 1 L All l£ 












specialist staff 

no specialist staf f t 


-.83/-. 13 


-.89/-. 12 


1.86/1.85 





.86/. 92 


Referral 












% DDS or SSDO referrals 


.01/-. 01 


.72/-. 62 


-.01/. 02 




-.18/. 58 


Use of Soecial Program 












% beneficiaries served in 
special program 


*.05/.05* 


2.39/1.54 


*.16/.07 




2.81/1.16 


intercept 




.10/5.27 


-18.42/- 


• 8.86 




R 2 


.307/. 235 


.460/ 


361 




R~ 




101/. 014 


.348/ 


.228 




F 


1.49/1.06 


4.11/2.73 




significance of F 




176/. 415 


.000/ 


.005 




n 




49/50 


71/71 




min n 




49/50 


71/71 





Notes : 

< 

* significant at .20 level or less -- two tailed test of significance for null hypothesis that coefficient 
is zero 



t base category for a dummy variable — category to which coefficients are compared 

X variable excluded from the regression: blind agency is excluded when only general agencies are being 
analyzed; with insubstantial handicaps is excluded when all agencies are being analyzed because of 
collinearity with blind agency -- see text; expenditure per client served is excluded from coverage and 
cost per SCA dependent variable regressions because of shared terms in the definitions -- see text. 
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Table 14 
Regressions; SSDI Clients ; 



Dependent Variable; Rehabilitation Rate 





General 


Agencies 


All Apcncies 


In 1 gender- ariablc 


coefficient 


t-valuc 


coefficient 


t-value 


FY75/FY76 


FY75/FY76 


FY75/FY76 


FY75/FY76 


Economic lir.vironmcnt 










unemployment rate 
average wages 


-.70/-. 09 
.058/. ISO* 


-.63/-. 05 
.60/1.86 


-.04/,fcj 
.016/.0S8 


-.45/. 51 
.^i:/1.27 


average beneiit 


^o/-.405 


-.72/-1.01 


-.061/-. 158 


-.26/-. 65 


VR Agency 










education agency 
umbrella agency* 
independent agency t 


-2.72/-2.34 
-7. 84/- 1.47 


-.47/-. 38 
-1.37/-. 29 


1.23/. 58 
-2. 18/. 06 


.26/. 13 
-.53/. 02 


blind agency, 
general agencyt 


X 


X 


*4 . .53/21 . 64* 


3.76/3.18 


large agency 
sma 1 1 agency t 


-4.59/-9.2S* 


-.96/-2.01 


♦-5.03/-6.12* 


-1.43/-1.90 


Expenditures 










expenditures per client served 
% expenditures on administra- 
tion 

* expenditures on counseling 
and placement 


-.0032/-. 0048 
. 12/- . 66 

.02/-. 07 


-.62/-1.17 
.19/-1.15 

.08/-. 35 


-.0015/-. 0005 

no/ j*3* 
• uy/ - « 

-.02/-. 12 


-1.00/-. 25 

• o 1/ - 1 • J* 

-.16/-. 90 


Staffing 










specialist staff 

no specialist stafft 


-5.48/-S.69 


-.90/-1.03 


♦-6.02/-3.15 


-1.57/-. 97 


Referral 










% DDS or S5D0 referrals 


.04/-. 03 


.34/-. 23 


.05/. 07 


.60/. 85 


Use of Special Program 










% beneficiaries served in 
special program 


-.06/-. 11 


-.42/-. 65 


*-.21/-.20* 


-1.68/-1.00 


Clients - Demographics 










% between 25 and 44 years old 
iiuci witn i «. or more years 
of education 

% male 

% white 


-.36/. 01 

- .06/- . 37 
.04/. 04 


-1.30/. 03 

-.20/- 1.07 
.37/. 33 


*-.28/.02 
.01/-.03 


-1.70/. 13 
.09/-. 36 


Clients - Economic Situation 










% with dependents 
% with earnings at referral 
% with public assistance at 
referral 


-.08/-. 22 
-.27/-. 20 
-.03/. 06 


-.35/-. 75 
-.64/-. 37 
-.21/. 49 


-.03/-. 14* 
*-.47/-.00 
-.09/. 06 


-.25/-1.74 
-2. 11/-. 01 
-.97/. 57 


Clients - Disability 










» with insubstantial handicaps 


.03/-. 07 


.08/-. 19 


X 


X 


Services 










\ services spent on diagnosis 
and evaluation 

% services spent on restora- 
tion 

% services spent on training 


-.19/-. 36 

.07/. 15 
.08/-. 05 


-.46/-. 86 

.24/. 64 
.24/-. 18 


-.11/-. 42* 

.08/. 18 
.09/. 01 


-.37/-1.61 

.52/1.24 
.74/. 04 



(continued) 
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Table 14 (cont.) 



Independent Variable 


General Agencies 


All Agencies 


coefficient t -value 


coefficient t- value 


FY75/FY76 FY75/FY7C) 


FY75/FV76 FY75/FY76 


'" 

Time in Process 

months in referral 
months from referral to 
closure 


- .62/-1 . 02 - .49/- .81 
.22/-. 13 .45/-. 30 


- .63/1 . 1 / w - .89/- 1 . 5« 
-.14/-. IS -.60/-. 61 


intercept 


127.S0/183.54 


113.40/126.99 


R 


Ace./ cTO 
.456/ . 5 


t- At. 1 ml 
• C»4o/ . 70J 


R" 


-.045/. 064 


.468/. 546 


F 


.90/1.14 


3.68/4.51 


significance of F 


.591/. 378 


.000/. 000 


n 


51/59 


74/73 


min H 


51/49 


74/71 



Notes : 

* significant at .20 level or less -- two tailed test of significance for null hypothesis that coefficient 
is zero 

t base category for a dummy variable -- category to which coefficients are compared 

X variable excluded from the regression: blind agency is excluded when only general agencies are being 
analyzed; * with insubstantial handicaps is excluded when all agencies ai» being analyzed because of 
collinearity with blind agency -- see text; expenditure per client served is excluded from coverage and 
cost per SGA dependent variable regressions because of shared terms in the definitions -- see text. 
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Table 15 
Regressions; SSI Clients; 
D ependent Variable: Rehabilitation Rate 





General 


Agencies 


All Agencies 




coefficient 


t-va luc 


coefficient 


t-value 


Independent Variable 


FY75/FY76 


FY75/FY76 


FY75/FY76 


FY75/FY76 


Icononie Environment 

unemployment rate 
average wages 


»-2.95/-2.75* 
'-.129/. 098 


-2.1S/-1.56 
-1.62/1.30 


*-2.22/-1.71* 
-.046/. 027 


-2.09/-1.31 
-.69/. 48 


Benefit Levels 
average benefit 


.018/. 046 


.15/. 30 


-.019/. 106 


-.19/. 97 


VR Agency 

education agency 
umbrella agency 
independent agency t 

blind agency 
general agency t 

large agency 
small agency t 


M0.74/-2.82 
1.13/2.20 

X 

*-9. 09/^10. 12* 


1.70/-. 51 
.19/. 37 

X 

-1.91/-2.07 


*7.02/3.37 
2.52/6.86* 

♦13.90/27.41* 

♦-6.34/-2.40 


1.36/. 67 
.61/1.66 

2.04/5.78 

-1.57/-. 61 


Expenditures 

expenditures per client served 
\ expenditures on administra- 
tion 

\ expenditures on counseling 
and placement 


-.0001/. 0011 
.25/-. 21 

*.43/-.08 


-.01/. 27 
.41/-. 30 

1.63/-. 55 


-.0024/. 0012 
.06/-. 94* 

*.29/-.10 


-1.00/. 71 
.18/-2.44 

1.90/-. 79 


Staffing 

specialist staff 

no specialist staff T 


♦-13.60/-1.03 


-2.23/-. 17 


-3.il/-6.96* 


-.74/-1.5S 


Referral 
% DDS or SSDO referrals 


.09/. OS 


.82/. 88 


-i09/.14* 


-1.09/1.56 


Use of Snecial Program 

% beneficiaries served in 
special program 


.01/-. 07 


.08/-. 47 


*-.22/-.09 


-1.86/-. 68 


Clients - Demographic 

% between 25 and 44 years old 
and with 12 or more years 
of education 

\ male 

% white 


-.04/-. 09 

-.06/-. 26 
-.09/- .06 


-.20/-. 33 
-.28/-. 92 

Aft / £R 


-.01/. 22* 

.04/-. 43* 
.01/-. 03 


-.11/1.54 

.31/-2.91 
.10/-. 32 


Clients - Economic Situation 

% with dependents 
% with earnings at referral 
% with public assistance at 
referral 


-.11/. 08 

.48/-. 33 
-.12/-. 02 


-.40/. 32 

.90/-. 85 
-.93/-. 12- 


*.29/-.07 

.02/. 23 
-.03/. 06 


1.65/-. 44 
.11/1.05 
-.37/. 69 


Clients - Disability 
\ with insubstantial handicaps 


-.11/-. 19 


-.42/-. 67 


X 


X 


Services 

% services spent on diagnosis 

and evaluation 
% services spent on restoration 
*• services spent on training 


.26/. 04 

.12/. 26 
.39/. 20 


.67/. 11 

.33/. 85 
1.08/. 76 


-.05/-. 18 

-.27/. 36* 
.00/. 10 


-.19/-. 77 

-1.22/1.59 
.03/. 59 
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Table 15 (cont.) 



Independent Variable 


General Agencies 


All 


Agencies 


coefficient t-valuc 


coefficient 


t-valuc 


FY75/FY76 FY75/FY76 


FY75/FY76 


FY75/FY76 


Tine in Process 

months in referral 
months from referral to 
closure 


-.23/. 43 -.22/. 42 
-.32/- .30 -.73/-. 79 


.63/1.13* 
.01/-. 30 


.77/1.50 
. 03/- 1 .03 


intercept 


103.07/72.99 




98.98/64.78 


*> 


.625/. 490 




.645/. 645 


R 2 


.233/-. 019 




.455/. 452 


F 


1.59/. 96 




3.40/3.33 


significance of F 


.134/. 537 




.000/. 000 


n 


49/50 




71/71 


nin n 


48/49 




70/69 



Notes : 

* significant at .20 level or less two tailed test of significance for null hypothesis that coefficient 
is zero 

t base category for a dummy variable — category to which coefficients are compared 

X variable excluded from the regression: blind agency is excluded when only general agencies are being 
analysed; *• with insubstantial handicaps is excluded when all agencies are being analyzed because of 
collinearity with blind agency see text; expenditure per client served is excluded from coverage and 
cost per SGA dependent variable regressions because of shared terms in the definitions -- see text. 
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Table 16 
Regressions; SSDI Clients; 
Dependent Variable: Cost Per SGA 





General Agencies 


All 


Ac* on cios 


Independent Variable 


coefficient 


t-value 


coefficient 


t-value 


FY75/FY76 


FY7S/FY76 


FY75/FY76 


FY75/FY76 


Economic Environment 

unemployment rate 
average wages 


♦1131/1295 
♦-71/-73 


1.93/. 45 
-1.48/-. 56 


439/-742 
-8/-67 


.66/-. 43 
-.13/-. 71 


Benefit Levels 
average benefit 


73/-449 


.52/-. 79 


12/7 


.07/. 02 


VR Agency 

education agency 
umbrella agency 
independent agency t 

blind agency 
general agency t 

large agency 
small agency t 


1701/7680 
-2674/-5154 

X 

1157/5325 


.59/. 83 
-.94/-. 69 

X 

.47/. 85 


♦-48S0/5329 
-2414/749 

3.796/9789 

*-3686/3445 


-1.38/. 87 
-.77/. 16 

.74/1.04 

-1.35/. 77 


ExDenditures 

expenditures per client 
served 

% expenditures on administra- 
tion 

% expenditures on counseling 
and placement 


x 

*642/1283* 
*-198/89 


X 

2.02/1.49 
-1.55/. 28 


X 

♦441/546 
♦220/102 


X 

2.07/1.23 
2. 19/. 53 


Staffing 

■ i\ 

specialist staff 

no specialist staff t 


-1433/6444 


-.46/. 82 


3607/4264 


1.19/. 94 


Referral 
• DOS or SSDO referrals 


59/52 


1.05/. 31 


-48/8 


-.78/. 07 


Use of Snecial Program 

% beneficiaries served in 
special program 


-31/-17 


-.40/-. 07 


-79/1S1 


-.83/. 87 


Clients - Demographic 

» between 25 and 44 years 
. old and with 12 or more 

years of education 
• male 
% white 


♦324/522 

-131/-288 
-39/115 


2.58/1.10' 

-.89/-. 56 
-.83/. 77 


-9/1 

-157/82 
43/61 


-.07/. 00 

-1.02/. 45 
• oy/ .b4 


Clients - Economic Situation 

\ with dependents 
% with earnings at referral 
% with public assistance 
at referral 


39/-1S4 
♦539/2144* 
-38/449* 


.31/-. 36 
2.56/2.67 — 
-.60/2.28 


♦152/96 

245/356 
♦128/364* 


1.61/. 62 
-.01/1.23 
1.73/2.67 


Clients - Disability 

-4 with insubstantial 
handicaps 


213/-137 


1.02/-. 25 


X 


X 



(continued) 
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Table 16 (cont.) 








General Agencies 


All Agencies 




coefficient t- value 


coefficient . t-value 


FY7S/FY76 FY7S/FY76 


FY75/FY76 FY7S/FY76 


Services 






% services spent on diag- 
nosis and evaluation 

\ services spent on restora- 
tion 

% services spent on training 


-49/-305 -.23/-. 51 
-98/391 -.66/1.09 
•-360/146 -2. 28/. 40 


68/-232 .30/-. 63 
-53/-38 -.42/-. 19 
-52/-12 -,54/-. 07 


Tine in Process 






months in referral 
months from referral to 
closure 


170/-3394* .26/- 1.80 
-125/-175 -.50/-. 28 


M418/-1200 2.60/-1.15 
78/-196 .43/-. 57 


intercept 


13743/81927 


-1515/-3761 


R 2 


■ V+Hf .SOI 


< jJl/ • J-4U 


R 2 


.341/. 003 


.316/. 017 


F 


2.12/1.01 


2.46/1.05 


significance of F 


.031/. 491 


.004/. 426 


n 


51/50 


74/73 


min n 


51/49 


74/71 



Notes: 



* significant at .20 level or less — two tailed test of significance for null hypothesis that coefficient 
is zero 

t base category for a dummy variable — category to which coefficients are compared 

X variable excluded from the regression: blind agency is excluded when only general agencies are being 
analyzed; • with insubstantial handicaps is excluded when all agencies are being analyzed because of 
collincarity with blind agency --see text ; expenditure per client served is excluded from coverage and 
cost per SGA dependent variable regressions because of shared terms in the definitions --see text. 
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Table 17 
Regressions; SSI Clients; 
Dependent Variable: Cost Per SGA 





General 


Agencies 


All 


Agencies 


Independent Variable 


coeffic icnt 


t-valuc 


coefficient 


t-valuc 




FY75/FY76 


FY75/FY76 


FY76/FY76 


FY75/FY76 


Econonic Environment 

unemployment rate 
average wages 


-3841/-3163 
353/-S 


-.40/-. 47 
.63/-. 06 


-2693/-374 
♦62S/-34 


-.42/-. 19 
1.57/-. 40 


Benefit Levels 
average benefit 


-414/247 


-.50/. 90 


241/-22 


.39/-. 14 


VR Agency 

education agency 
umbrella agency 
independent agency t 

blind agency 
general agency t 

large agency 
small agency t 


-38490/10566 
♦-59486/1144 

X 

-16106/-1862 


-.86/. 99 
-1.52/. 10 

X 

-.49/-. 20 


♦-58445/8405 
-31244/-4904 

-12138/2118 

5140/-I72 


-1.91/1.11 
-1.26/-. 79 

-.30/. 20 

.21/-. 08 


Exuenditurcs 

expenditure per client served 
\ expenditures on administra- 
tion 

*i expenditures on counseling 
and placement 


X 

♦6713/-.764 
-264/-107 


X 

1.60/-. 55 
-.16/-. 37 


X 

1800/461 
-937/-46 


X 

.96/. 79 
-1.01/-. 24 


Staffing 

specialist staff 

no specialist staff t 


♦58727/3019 


1.52/. 27 


30037/2838 


1.2C/.42 


Referral 
\ 0D5 or SSDO referrals 


225/160 


.29/. 89 


-268/169 


-.55/1.28 


Use of Special Program 

* beneficiaries served in 
special program 


-261/92 


-.23/. 30 


-6S5/-65 


-.91/-. 32 


Clients - Demographics 

% between 25 and 44 years 
old and with 12 or more 
years of education 

% male 

% white 


815/-153 

♦-3055/-403 
-86/204 


.57/-. 33 

-1.88/-. 74 
-.09/1.08 


-738/-125 

-957/89 
306/196* 


-1.08/-. 57 

-1.17/. 40 
.51/1.54 


Clients - Economic Situation 

* with dependents 

% with earnings at referral 


*-2685/-459 
*S211/523 


-1.44/-. 96 
1.45/. 71 , 


*-1373/-237 
846/506* 


-1.31/-1.05 
.67/1.49 


* with public assistance 
at referral 


29/5 


.03/:02- — 


*738/-82 


-1:38/-. 58 


Clients - Disability 

% with insubstantial handi- 
caps 


-716/544 


-.39/1.00 


X 


X 
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Table 17 (cont.) 




General Agencies 


All Agencies 




coefficient t-value 


coefficient t-value 


Independent Variable 


FY75/FY76 FY75/FY76 


FY75/FY76 


FY75/FY76 


Services 








% services spent on diagnosis 
and evaluation 

% services spent on restora- 
tion 

\ services spent on training 


♦-6169/-547 -2. 42/-. 88 

-2196/ o72 - .07/ .65 
♦-4783/-190 -2. 16/-. 38 


-1769/-188 
1348/307 
-915/-25 


-1.28/-. 54 

1.02/. 94 
-1.00/-. 10 


time in rroccas 








months in referral 
months from referral to 
intake 


-1463/-796 -.48/-1.10 


-4784/1315 
809/-85 


-.97/1,10 
.57/-. 19 


intercept 


648498/25217 


48921/9931 


R* 


,545/. 389 


.325/. 289'" 


— -> 
R- 


,110/-. 173 




013/-. 074 


F 


1.25/. 69 




.96/. 80 


significance of F 


.294/. 810 




526/. 718 


n 


49/50 




71/71 


min n 


48/49 




70/69 



Notes : 

* significant at .20 level or less -- two tailed test of significance for null hypothesis that coefficient 
is zero • 

t base category for a dummy variable -- category to which coefficients are compared 

X variable excluded from the regression: blind agency is excluded when only general agencies are being 
analyzed; \ with insubstantial handicaps is excluded when all agencies are being analyzed because of 
collinearity with blind agency -- see text; expenditure per client served is excluded from coverage and 
cost per 5GA dependent variable regressions because of snared terms in the definitions — see text. 
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Table 18 
Regressions; SSDI Clients; 
Dependent Variable: % SGA 





Hon o 1 


Agencies 


All Agencies 


Independent Variable 


coefficient 


t- value 


coefficient 


t-value 


FY75/FY76 


FY75/FY76 


FY75/FY76 


FY75/FY76 


Economic Environment 










unemployment rate 
average wages 


-.37/-. 65 
.239/-. 030 


-.29/-. 22 
2.14/* . 20 


-.80/1.35 
.090/. 046 


-.83/. 73 
1.00/. 45 


Benefit Levels 










average benefit 


-.361/. 408 


-1.02/. 67 


-.015/-. 166 


-.58/-. 46 


VR Agency 










education agency 
umbrella agency 
independent agency t 


3.99/3.66 
-4.40/5.06 


-.60/. 39 
-.66/. 64 


2.50/1.53 
-1.78/1.58 


.46/. 23 
-.38/. 29 


blind agency 
general agency t 


X 


X 


*-10.28/-9.78 


-1.40/-. 97 


large agency 
small agency t 


4.38/-3.8S 


.79/-. 55 


1.33/-2.58 


.33/-. 54 


Exoendi tures 










expenditures per client served 
% expenditures on administra- 
tion 

% expenditures on counseling 
and placement 


*-. 0094/. 0044 
-.01/-. 00 

-,03/. 16 


-1.56/. 70 
-.02/-. 69 

-.12/. 48 


-.0010/. 0030 
.08/-. 62 

-.01/. 01 


-.55/1.11 
.27/-1.29 

-.09/. 06 


Staffing 










specialist staff 

no specialist staff f " 


-3. 34/-. 99 


-.47/-. 12 


-4.92/-3.71 


-1.13/-. 77 


Referral 










\ DD5 or S5D0 referrals 


*-.20/-.06 


-1.58/-. 37 


-.11/-. 08 


-1.17/-. 63 


\J J C w L. V c w X u jl r I U c 1 










% beneficiaries served in 
special program 


-.12/. 16 


-.68/. 60 


-.09/. 05 


-.66/. 26 


Clients - Demographics 










\ between 25 and 44 years 
old and with 12 or more 
years of education 

\ male 

% white 


*.73/.09 

*.76/.48 
*.2y/-.22 


2. 30/. 17 

2.32/. 91 
2.43/-1.26 


*.S8/.26 

*.45/.l9 
*.14/-.03 


3.11/1.20 

2. 03/. 96 
1.50/-. 26 


Clients - Economic Situation 










\ with dependents 
% with earnings at referral 
% with public assistance at 
referral 


.19/. 53 
-.17/-. 89 
.08/-. 32* 


.70/1.15 
-•33/-UD94.1- 
.S9/-1.61 


.11/-. 10 
.18/-. 02 
.03/.. 17 


.81/-. 59 
', -.71/-. 05 
.31/-1.18 


Clients - Disability 










\ with insubstantial handi- 
cap 


-.00/. 59 


-.01/1.07 


X 


X 
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Table 18 (cont.) 







General Agencies 


All Agencies 






coefficient 




t-value 


coefficient 


t-value 


Independent Variable 




FY75/PY76 




FY75/FY76 


FY75/FY76 


PY75/FY76 


Services 














• services spent on 


diagnosis 


.19/. 25 




.38/. 39 


-.18/. 20 


-.57/. 52 


and evaluation 














*• services spent on 


rcstora- 


'-.IS/-. 66* 




-.S4/-1.S0 


-.11/-. 26 


-.58/-1.20 


tion 














% services spent on 


training 


-.24/-. 25 




-.64/-. 62 


-.11/-. 07 


-.82/-. 38 


Tine in Process 














months in referral 




1.64/4.48* 




1.12/2.34 


♦1.29/1.57* 


1.61/1.40 


months from referral 


to 


• if Of • £ I 




1.66/. 41 


♦.38/. 54* 


1.37/1.46 


closure 














intercept 






8.12/-77.32 


-10.46/54. 


80 


R 






.630/. 439 


.591/. 336 


R- 






.289/-. 


123 


.391/-. 011 


F 






1.85/. 78 


2. 95/. 97 




significance of F 






.064/. 7 


25 


.001/. 52 


0 


n 






51/50 




74/73 




tnin n 






51/49 




74/71 





Notes : 

* significant at .20 level or less -- two tailed test of significance for null hypothesis that coefficient 
is zero 

t base category for a dummy variable — category to which coefficients are compared 

X variable excluded from the regression: blind agency is excluded" when only general agencies are being 
analyzed; \ with insubstantial handicaps is excluded when all agencies are being analyzed because of 
collincarity with blind agency — see text; expenditure per client served is excluded from coverage and 
cost per SGA dependent variable regressions because of shared terms in the definitions --see text. 
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Tabic 19 
' Regressions; SSI Clients; 
Dependent Variable: % SGA 





General 


Agencies 


All Agencies 




coefficient 


t-value 


coefficient 


t-value 


Independent Variable 


FY75/FY76 


FY75/FY76 


. FY75/FY76 


FY75/FY76 


Economic Environment 

unemployment rate 
average wages 


2.09/2.30* 
.048/. 038 


1.23/1.59 
.48/. 43 


. 36/1 . 23 
.025/. 033 


.27/ .88 
.29/. 54 


Benefit Lev: Is 
average benefit 


-.047/-. 030 


-.32/-. 16 


-.074/. 067 


-.56/. 57 


VR Agency 

education agency 
umbrella agencv 
independent agencyt 

blind agency 
general agencyt 

large agency 
small agencyt 


-2.35/-1.45 
-4.25/3. 18 

X 

-6.18/-2.75 


-.30/-. 23 
-.55/. 45 

X 

-1.04/-. 48 


2.10/-1.41 
3.66/4.98 

-4.59/4.23 

-1.94/-3.76 


.32/-. 26 
.69/1.12 

-.53/. 54 

-.37/-. 89 


Exnpndi tni*p^ 

expenditure per client served 
* expenditures on administra- 
tion 

*• expenditures on counseling 
and placement 


.0059/. 0042 
-.00/-. 24 

-.24/. 232* 


.69/. 88 
-.00/-. 28 

-.72/1.35 


-.0020/-. 0015 
-.15/-. 09 

-.15/. 32* 


-.65/-. 81 
-.38/-. 21 

-.76/2.31 


Staffing 

specialist staff 

no specialist stafft 


*-11.97/3.39 


-1.56/. 49 


-5.43/-4.20 


-1.01/-. 89 


Dp f n r r 1 
f\C I IT 1 1 <J 1 

% DDS or SSDO referrals 


.05/. 00 


.38/. 03 


.02/. 04 


.20/. 43 


Use of Special Program 

\ beneficiaries served in 
special program 


-.11/. 01 


-.49/. 05 


-.06/ . 02 


- .36/. 13 


Clients - Demographics 

% between 25 and 44 years 
old and with 12 or more 
years of education 

% male 

\ white * 


.33/. 24^ 

*.48/.59* 
-.18/-. 31* 


1.25/. 77 

1 .67/1.78 
-1.08/-2.49 


*.27/.15 

$.68/. 75* 
*-.33/-.22* 


1.86/. 98 

3.94/4.73 
-2.S8/-2.44 


Clients - Economic Situation 

% with dependents 

* with earnings at referral 

* with public assistance 
at referral 


.06/. 04 
-.53/-. 17 
-.02/. 17 


.16/7tT~~ 
-.79/-. 38 
-.15/1.05 


-.15/. 18 

.33/-. 27 
-.12/. 11 


-.66/1.11 
1.22/- 1.13 
-1.06/1.14 


Clients - Disability 

% with insubstantial handi- 
caps 


.10/. 09 


.32/. 27 


X 


X 
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Table 19 (cont.) 





General Agencies 


All 


Agencies 




Independent Variable 


coefficient 


t-value 


coefficient 




t-value 


FY75/FY76 


FY75/FY76 


FY75/FY76 


FY75/FY76 


Services 












^ services spent on diagnosis 
- and evaluation 


.62/. 02 


1.27/. 06 


.36/. 12 


1 


.18/. 45 


% services spent on restora- 
tion 


.03/. 06 


.06/. 16 






.92/. 05 


services spent on training 


.16/. 20 


.35/. 65 


-.01/. 07 




.05/. 37 


Time in Process 












months in referral 
months from referral to 
closure 


.95/-. 09 
♦.98/1.46* 


.72/-. 07 
1.77/3.29 


-.03/-. 22 
*. 88/1. 06* 


-.02/-. 03 
2.92/3.39 


intercept 


-15 


.69/-66.S0 


34.97/-50.67 




.553/. 671 


.560/. 720 






.039/. 341 


.326/. 567 




f 


1 


.19/2.04 


2. 


39/4.71 




significance of F 




343/. 044 


.006/. 000 




n 




49/50 




71/71 




min n 




48/49 




70/69 





Notes : 

* significant at .20 level or less two tailed test of significance for null hypothesis that coefficient 
is zero 

t base category for a dummy variable — category to which coefficients are compared 

X variable excluded from the regression: blind agency is excluded when only general agencies are being 
analyzed; •> with insubstantial handicaps is excluded when all agencies are being analyzed because of 
collinearity with blind agency see text; expenditure per client served is excluded from coverage and 
cost per SGA dependent variable regressions because of shared terms in the definitions — see text. 
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For the regressions using coverage as the dependent variable, none 
of the factors occurring after referral (such as the demographics of 26 
closures) are used, because such factors are temporally and thus causally 
after the decision to serve. Expenditure per client served is excluded 
from the coverage and cost-per-SGA regressions because of shared terms in 
the definitions, number of clients served for coverage, and total expendi- 
tures for cost per SGA. Such shared terms lead to artificial coorelations. 
The percent with insubstantial employment handicaps measure is excluded 
from the regressions for all agencies, because of colinearity with the 
blind agency (0,1) variable. 

The regressions are disappointing in several respects. First, the 
explanatory power of several of the regressions is poor (e.g., coverage 
for SSDI clients). Second, the results vary widely from year to year, 
from general agency to all agencies, and from client type to client type 
(from SSDI to SSI). Sometimes these results conflict. For few of the 
regressions do many statistically significant factors emerge, and, very 
few patterns emerge. 

. Rather than discuss each regression separately and rather than 
speculate on the disappointing results, the few patterns that do emerge 
are discussed. . 

For rehabilitation rate, the factors explaining performance all- 
fall into the category of characteristics of the agency and of the selection 
of clients. Blind agencies have higher rehabilitation rates, as do 
smaller agencies. Agencies spending money on specialist staff have lower 
rehabilitation rates. And agencies taking a larger proportion of benefici- 
aries into the special program have lower rehabilitation rates. None of 
these relations has implications for policy, since rehabilitation rate 
itself is not related to the primary goal of the special program, i.e., 
termination or reduction in benefits. 

The more interesting patterns exist for the regressions using as the 
dependent variable % SGA. Clearly, the demographic characteristics of 
clients are related to performance in this measure. The traditional vari- 
ables of age, education, sex, and race are all related to performance in 
the ways usually found in labor participation and income studies (except 
for the negative effect of percent white for SSI clients) . These findings 
also agree with the micro results in Berkowitz, et al . 
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Based on such a finding, a policy of creaming is usually suggested. 
There are, of course, several problems with suggesting such a policy. <, 
First is the problem of discrimination, given that sex, age, and race 
are involved. Second is the question of whether there is any more room 
for creaming. That is, creaming may be so pervasive now that there is no 
more slack for further expansion of creaming. To empirically examine the 
slack for creaming, demographics for all the eligible beneficiaries in 
the state who could meet the SSC would be needed. However, these data 
are not available. Second best, date on 26 closures could be used. A 
look at Tables 20 and 21 indicate that there is more room for creaming. 
For example, 25-SP% of the 26 closures are in the prime age/education 
group; it is unlikely that this is all of the possible clients in this 
demographic gTOup. 

A third question is more telling, however. Is creaming really in 
the interests of the special program? Creaming will certainly lead to a 
higher cost-benefit ratio (B -e- C). However, the more important criterion 
for the special program is discounted net present value (B - C) , since 
this criterion addresses the net outflow of monies from the Trust Fund, 

Given the above, the following strategy is suggested. Clients in the 
above demographic groups, more likely to succeed, would still be served, 
but with less attention, since they will likely succeed anyway. Clients 
in the demographic groups less likely to succeed would be given special 
attention, to try to bring them to employment and thus a reduction in 
benefits or termination. Such a strategy might lead to a smaller % 
SGA and CB ratio, but also to a larger savings to the Trust Fund. 

Another finding from the % SGA regressions is that time in 
process is positively related to % SGA, While there may be differing 
interpretations of this finding, it seems that the intensity and duration 
of the service process is being measured. That is, .controls for other 
possible explanations are included in the analysis, e.g. , for expenditures, 
services, and demographics. 
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Table 20 

Demographic Characteristics of SSDI Clients 





% 25-44 years 

old with 
12+ education 


% male 


% white 


FY75/FY76 


FY75/FY76 


FY75/FY76 


minimum 


14.9/0' 


44.9/0 


1.1/0 


maximum 


66.7/100.0 


100.0/100.0 


100.0/100.0 


mean 


38.5/25/8 


73.9/54.1 


84.4/75/9 


median 


37.6/25.0 


73.1/52.9 


90.8/83.7 


number of agencies 


74/70 


-74/71 


74/71 



Table 21 

Demographic Characteristics of SSI Clients 





% 25-44 years 

old with 
12+ education 


% male 


% white 


FY75/FY76 


FY75/FY76 


FY75/FY76 


minimum 
maximum 
mean 
median 

number of agencies 


0/0 
100.0/100.0 
28.0/25.8 
26.2/25.0 
71/70 


0/0 
76.2/100.0 
49.4/54.1 
51.9/52.9 
71/71 


0/0 
100.0/100.0 
76.8/75.9 
83.3/83.7 
71/71 
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Cross-Sectional and Time Series Analyses 

The example presented is a cross-sectional analysis, done for two 
different time periods (1975 and 1976)- As the standards system operates 
over time, a time series of data will also be available for each state VR 
agency. As such, analysis of agency-level data will move from cross-sectional 
analysis to time series analysis for a particular agency and for the nation 
as a whole, and to combined time-series cross-sectional analysis. While 
the statistical methodology for these analyses is more involved*, the basic 
analytical approach used in the example will still apply. 



See Balestra and Nerlove, Box and Jenkins, and Glass, Willson, and 
Gottman for examples of the types of time-series analyses and of time- 
series cross-sectional analyses involved. 
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